Note: This is test shard 1 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from AdminCliTest
[ RUN ] AdminCliTest.RebalancerReportOnly
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 08:10:39.991742 6369 test_util.cc:268] Using random seed: 95428491
I20240501 08:10:40.003738 6369 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:10:40.003824 6369 ts_itest-base.cc:113] --------------
I20240501 08:10:40.003870 6369 ts_itest-base.cc:114] 5 tablet servers
I20240501 08:10:40.003932 6369 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:10:40.003994 6369 ts_itest-base.cc:116] --------------
2024-05-01T08:10:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:10:40Z Disabled control of system clock
I20240501 08:10:40.029165 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:33021
--webserver_interface=127.6.56.126
--webserver_port=0
--builtin_ntp_servers=127.6.56.84:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.56.126:33021 with env {}
W20240501 08:10:40.361752 6381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:40.362071 6381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:40.362207 6381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:40.369691 6381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:10:40.369815 6381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:40.369865 6381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:10:40.369951 6381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:10:40.379511 6381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.56.126:33021
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.56.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:40.380944 6381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:40.382680 6381 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:40.394596 6386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.395341 6387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.396023 6389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:40.397009 6381 server_base.cc:1034] running on GCE node
I20240501 08:10:40.397905 6381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:40.399533 6381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:40.400758 6381 hybrid_clock.cc:648] HybridClock initialized: now 1714551040400692 us; error 44 us; skew 500 ppm
I20240501 08:10:40.401101 6381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:40.409672 6381 webserver.cc:460] Webserver started at http://127.6.56.126:33117/ using document root <none> and password file <none>
I20240501 08:10:40.410749 6381 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:40.410929 6381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:40.411432 6381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:40.415700 6381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "01b6ec94b2c740958acd7d1c03b990a3"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-f1nk"
I20240501 08:10:40.416646 6381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "01b6ec94b2c740958acd7d1c03b990a3"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-f1nk"
I20240501 08:10:40.421990 6381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 08:10:40.425720 6395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.427207 6381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:40.427405 6381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
uuid: "01b6ec94b2c740958acd7d1c03b990a3"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-f1nk"
I20240501 08:10:40.427613 6381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:40.444666 6381 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:40.445573 6381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:40.445859 6381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:40.465349 6381 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.126:33021
I20240501 08:10:40.465384 6446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.126:33021 every 8 connection(s)
I20240501 08:10:40.466820 6381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:10:40.470621 6447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:40.476580 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 6381
I20240501 08:10:40.476828 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:10:40.480051 6447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3: Bootstrap starting.
I20240501 08:10:40.484091 6447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:40.485186 6447 log.cc:824] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:40.487639 6447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3: No bootstrap required, opened a new log
I20240501 08:10:40.494450 6447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6ec94b2c740958acd7d1c03b990a3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 33021 } }
I20240501 08:10:40.494859 6447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:40.494985 6447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b6ec94b2c740958acd7d1c03b990a3, State: Initialized, Role: FOLLOWER
I20240501 08:10:40.495633 6447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6ec94b2c740958acd7d1c03b990a3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 33021 } }
I20240501 08:10:40.495834 6447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:40.495955 6447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:40.496098 6447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:40.499693 6447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6ec94b2c740958acd7d1c03b990a3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 33021 } }
I20240501 08:10:40.500214 6447 leader_election.cc:302] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01b6ec94b2c740958acd7d1c03b990a3; no voters:
I20240501 08:10:40.501055 6447 leader_election.cc:288] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:40.501256 6452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:40.502182 6452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [term 1 LEADER]: Becoming Leader. State: Replica: 01b6ec94b2c740958acd7d1c03b990a3, State: Running, Role: LEADER
I20240501 08:10:40.502794 6452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6ec94b2c740958acd7d1c03b990a3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 33021 } }
I20240501 08:10:40.503762 6447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:40.505502 6454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b6ec94b2c740958acd7d1c03b990a3. Latest consensus state: current_term: 1 leader_uuid: "01b6ec94b2c740958acd7d1c03b990a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6ec94b2c740958acd7d1c03b990a3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 33021 } } }
I20240501 08:10:40.505816 6454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:40.506328 6453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01b6ec94b2c740958acd7d1c03b990a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b6ec94b2c740958acd7d1c03b990a3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 33021 } } }
I20240501 08:10:40.506600 6453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:40.509948 6457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:40.517271 6457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:40.527707 6457 catalog_manager.cc:1347] Generated new cluster ID: e18aed7f558446a19ebc934f7311f9c8
I20240501 08:10:40.527863 6457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:40.542637 6457 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:40.543905 6457 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:40.555981 6457 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 01b6ec94b2c740958acd7d1c03b990a3: Generated new TSK 0
I20240501 08:10:40.556955 6457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:40.573457 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.65:0
--local_ip_for_outbound_sockets=127.6.56.65
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--builtin_ntp_servers=127.6.56.84:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240501 08:10:40.928099 6471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:40.928468 6471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:40.928628 6471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:40.936918 6471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:40.937172 6471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.65
I20240501 08:10:40.946308 6471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.65
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:40.947692 6471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:40.949491 6471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:40.962644 6477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.963230 6476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:40.964042 6479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:40.964785 6471 server_base.cc:1034] running on GCE node
I20240501 08:10:40.965512 6471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:40.966665 6471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:40.967907 6471 hybrid_clock.cc:648] HybridClock initialized: now 1714551040967839 us; error 46 us; skew 500 ppm
I20240501 08:10:40.968254 6471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:40.970649 6471 webserver.cc:460] Webserver started at http://127.6.56.65:44413/ using document root <none> and password file <none>
I20240501 08:10:40.971449 6471 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:40.971580 6471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:40.972402 6471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:40.979856 6471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ce6f663293db49408f32ff7a0c838bcb"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-f1nk"
I20240501 08:10:40.980779 6471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ce6f663293db49408f32ff7a0c838bcb"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-f1nk"
I20240501 08:10:40.986063 6471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20240501 08:10:40.989500 6485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:40.990856 6471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:40.991016 6471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ce6f663293db49408f32ff7a0c838bcb"
format_stamp: "Formatted at 2024-05-01 08:10:40 on dist-test-slave-f1nk"
I20240501 08:10:40.991194 6471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:41.008944 6471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:41.009814 6471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:41.010092 6471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:41.011054 6471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:41.012625 6471 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:41.012709 6471 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.012840 6471 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:41.012893 6471 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.046926 6471 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.65:43879
I20240501 08:10:41.046959 6597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.65:43879 every 8 connection(s)
I20240501 08:10:41.048244 6471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:10:41.052384 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 6471
I20240501 08:10:41.052620 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:10:41.056440 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.66:0
--local_ip_for_outbound_sockets=127.6.56.66
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--builtin_ntp_servers=127.6.56.84:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 08:10:41.063057 6598 heartbeater.cc:347] Connected to a master server at 127.6.56.126:33021
I20240501 08:10:41.063452 6598 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:41.064607 6598 heartbeater.cc:511] Master 127.6.56.126:33021 requested a full tablet report, sending...
I20240501 08:10:41.066965 6412 ts_manager.cc:194] Registered new tserver with Master: ce6f663293db49408f32ff7a0c838bcb (127.6.56.65:43879)
I20240501 08:10:41.068622 6412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.65:46909
W20240501 08:10:41.406607 6602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:41.406940 6602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:41.407068 6602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:41.414754 6602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:41.414952 6602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.66
I20240501 08:10:41.424515 6602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.66
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:41.425917 6602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:41.427701 6602 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:41.441442 6610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.441969 6608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.442903 6607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:41.443629 6602 server_base.cc:1034] running on GCE node
I20240501 08:10:41.444350 6602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:41.445340 6602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:41.446566 6602 hybrid_clock.cc:648] HybridClock initialized: now 1714551041446490 us; error 58 us; skew 500 ppm
I20240501 08:10:41.446870 6602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:41.449249 6602 webserver.cc:460] Webserver started at http://127.6.56.66:37757/ using document root <none> and password file <none>
I20240501 08:10:41.450086 6602 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:41.450225 6602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:41.450522 6602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:41.453210 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0e5ea129cce047a3bc01be742085bbd9"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-f1nk"
I20240501 08:10:41.453976 6602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0e5ea129cce047a3bc01be742085bbd9"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-f1nk"
I20240501 08:10:41.459177 6602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:10:41.462631 6616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.464138 6602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 08:10:41.464280 6602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0e5ea129cce047a3bc01be742085bbd9"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-f1nk"
I20240501 08:10:41.464502 6602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:41.484758 6602 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:41.486120 6602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:41.486404 6602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:41.487362 6602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:41.488919 6602 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:41.488991 6602 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.489082 6602 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:41.489125 6602 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.524000 6602 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.66:42971
I20240501 08:10:41.524037 6728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.66:42971 every 8 connection(s)
I20240501 08:10:41.525326 6602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:10:41.526240 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 6602
I20240501 08:10:41.526427 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:10:41.530433 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.67:0
--local_ip_for_outbound_sockets=127.6.56.67
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--builtin_ntp_servers=127.6.56.84:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 08:10:41.539178 6729 heartbeater.cc:347] Connected to a master server at 127.6.56.126:33021
I20240501 08:10:41.539486 6729 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:41.540359 6729 heartbeater.cc:511] Master 127.6.56.126:33021 requested a full tablet report, sending...
I20240501 08:10:41.541821 6412 ts_manager.cc:194] Registered new tserver with Master: 0e5ea129cce047a3bc01be742085bbd9 (127.6.56.66:42971)
I20240501 08:10:41.542582 6412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.66:43803
W20240501 08:10:41.891139 6732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:41.891489 6732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:41.891645 6732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:41.899577 6732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:41.899806 6732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.67
I20240501 08:10:41.909591 6732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.67
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:41.911067 6732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:41.912976 6732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:41.925577 6738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.926122 6740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:41.927002 6737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:41.927671 6732 server_base.cc:1034] running on GCE node
I20240501 08:10:41.928323 6732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:41.929311 6732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:41.930537 6732 hybrid_clock.cc:648] HybridClock initialized: now 1714551041930468 us; error 47 us; skew 500 ppm
I20240501 08:10:41.930862 6732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:41.933307 6732 webserver.cc:460] Webserver started at http://127.6.56.67:33197/ using document root <none> and password file <none>
I20240501 08:10:41.934027 6732 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:41.934141 6732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:41.934412 6732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:41.937028 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3774f5bbdb47493faeedddf0d02f647c"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-f1nk"
I20240501 08:10:41.937795 6732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3774f5bbdb47493faeedddf0d02f647c"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-f1nk"
I20240501 08:10:41.943415 6732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:10:41.946852 6746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.948164 6732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:41.948359 6732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3774f5bbdb47493faeedddf0d02f647c"
format_stamp: "Formatted at 2024-05-01 08:10:41 on dist-test-slave-f1nk"
I20240501 08:10:41.948530 6732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:41.974421 6732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:41.975306 6732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:41.975598 6732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:41.976598 6732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:41.978225 6732 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:41.978297 6732 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:41.978427 6732 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:41.978475 6732 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.011551 6732 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.67:39525
I20240501 08:10:42.011569 6858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.67:39525 every 8 connection(s)
I20240501 08:10:42.012938 6732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:10:42.019213 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 6732
I20240501 08:10:42.019426 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:10:42.022523 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.68:0
--local_ip_for_outbound_sockets=127.6.56.68
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--builtin_ntp_servers=127.6.56.84:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 08:10:42.026633 6859 heartbeater.cc:347] Connected to a master server at 127.6.56.126:33021
I20240501 08:10:42.027014 6859 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:42.027917 6859 heartbeater.cc:511] Master 127.6.56.126:33021 requested a full tablet report, sending...
I20240501 08:10:42.029502 6412 ts_manager.cc:194] Registered new tserver with Master: 3774f5bbdb47493faeedddf0d02f647c (127.6.56.67:39525)
I20240501 08:10:42.030650 6412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.67:44811
I20240501 08:10:42.072351 6598 heartbeater.cc:503] Master 127.6.56.126:33021 was elected leader, sending a full tablet report...
W20240501 08:10:42.368463 6862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:42.368768 6862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:42.368906 6862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:42.376557 6862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:42.376745 6862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.68
I20240501 08:10:42.386340 6862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.68
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:42.387686 6862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:42.389532 6862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:42.402808 6867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.403604 6868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.404394 6870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:42.405153 6862 server_base.cc:1034] running on GCE node
I20240501 08:10:42.405831 6862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:42.406821 6862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:42.408056 6862 hybrid_clock.cc:648] HybridClock initialized: now 1714551042407992 us; error 44 us; skew 500 ppm
I20240501 08:10:42.408445 6862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:42.411346 6862 webserver.cc:460] Webserver started at http://127.6.56.68:43587/ using document root <none> and password file <none>
I20240501 08:10:42.412062 6862 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:42.412205 6862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:42.412590 6862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:42.415179 6862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "3d65f90611c94a11bc46b045f9880f5f"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-f1nk"
I20240501 08:10:42.415938 6862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "3d65f90611c94a11bc46b045f9880f5f"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-f1nk"
I20240501 08:10:42.421180 6862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240501 08:10:42.424525 6876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.425976 6862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:10:42.426226 6862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "3d65f90611c94a11bc46b045f9880f5f"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-f1nk"
I20240501 08:10:42.426437 6862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:42.452018 6862 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:42.453001 6862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:42.453301 6862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:42.454290 6862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:42.455799 6862 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:42.455876 6862 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.456012 6862 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:42.456068 6862 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.495059 6862 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.68:43499
I20240501 08:10:42.495187 6988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.68:43499 every 8 connection(s)
I20240501 08:10:42.496726 6862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:10:42.501844 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 6862
I20240501 08:10:42.502034 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:10:42.505467 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.69:0
--local_ip_for_outbound_sockets=127.6.56.69
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--builtin_ntp_servers=127.6.56.84:33557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240501 08:10:42.510600 6989 heartbeater.cc:347] Connected to a master server at 127.6.56.126:33021
I20240501 08:10:42.510922 6989 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:42.512468 6989 heartbeater.cc:511] Master 127.6.56.126:33021 requested a full tablet report, sending...
I20240501 08:10:42.513983 6412 ts_manager.cc:194] Registered new tserver with Master: 3d65f90611c94a11bc46b045f9880f5f (127.6.56.68:43499)
I20240501 08:10:42.515192 6412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.68:49151
I20240501 08:10:42.545172 6729 heartbeater.cc:503] Master 127.6.56.126:33021 was elected leader, sending a full tablet report...
W20240501 08:10:42.857831 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:42.858158 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:42.858289 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:42.866340 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:42.866565 6992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.69
I20240501 08:10:42.876111 6992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:33557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:33021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.69
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:42.877499 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:42.879230 6992 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:42.892235 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.893285 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:42.893738 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:42.894623 6992 server_base.cc:1034] running on GCE node
I20240501 08:10:42.895368 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:42.896486 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:42.897717 6992 hybrid_clock.cc:648] HybridClock initialized: now 1714551042897651 us; error 44 us; skew 500 ppm
I20240501 08:10:42.898033 6992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:42.900506 6992 webserver.cc:460] Webserver started at http://127.6.56.69:45403/ using document root <none> and password file <none>
I20240501 08:10:42.901293 6992 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:42.901428 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:42.901753 6992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:42.904786 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3aab9eecade541a285688996ff3e72aa"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-f1nk"
I20240501 08:10:42.905656 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3aab9eecade541a285688996ff3e72aa"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-f1nk"
I20240501 08:10:42.911044 6992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20240501 08:10:42.915261 7006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.917003 6992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 08:10:42.917186 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3aab9eecade541a285688996ff3e72aa"
format_stamp: "Formatted at 2024-05-01 08:10:42 on dist-test-slave-f1nk"
I20240501 08:10:42.917380 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:42.950790 6992 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:42.951690 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:42.952005 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:42.953050 6992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:42.955139 6992 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:42.955238 6992 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.955374 6992 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:42.955428 6992 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:42.989544 6992 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.69:33851
I20240501 08:10:42.989590 7118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.69:33851 every 8 connection(s)
I20240501 08:10:42.990880 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:10:42.994921 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 6992
I20240501 08:10:42.995129 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.AdminCliTest.RebalancerReportOnly.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:10:43.005190 7119 heartbeater.cc:347] Connected to a master server at 127.6.56.126:33021
I20240501 08:10:43.005543 7119 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:43.006397 7119 heartbeater.cc:511] Master 127.6.56.126:33021 requested a full tablet report, sending...
I20240501 08:10:43.007738 6412 ts_manager.cc:194] Registered new tserver with Master: 3aab9eecade541a285688996ff3e72aa (127.6.56.69:33851)
I20240501 08:10:43.008570 6412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.69:38555
I20240501 08:10:43.012045 6369 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:10:43.033597 6859 heartbeater.cc:503] Master 127.6.56.126:33021 was elected leader, sending a full tablet report...
I20240501 08:10:43.035228 6411 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39562:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 08:10:43.075937 6533 tablet_service.cc:1467] Processing CreateTablet for tablet 48c944814a764167a88aa3b891fbb952 (DEFAULT_TABLE table=TestTable [id=db402601d54a4cd29f8f41bf309d30f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:43.075937 7054 tablet_service.cc:1467] Processing CreateTablet for tablet 48c944814a764167a88aa3b891fbb952 (DEFAULT_TABLE table=TestTable [id=db402601d54a4cd29f8f41bf309d30f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:43.077491 7054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c944814a764167a88aa3b891fbb952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:43.077534 6533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c944814a764167a88aa3b891fbb952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:43.081887 6664 tablet_service.cc:1467] Processing CreateTablet for tablet 48c944814a764167a88aa3b891fbb952 (DEFAULT_TABLE table=TestTable [id=db402601d54a4cd29f8f41bf309d30f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:43.083604 6664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48c944814a764167a88aa3b891fbb952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:43.089507 7137 tablet_bootstrap.cc:492] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa: Bootstrap starting.
I20240501 08:10:43.093163 7138 tablet_bootstrap.cc:492] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb: Bootstrap starting.
I20240501 08:10:43.093825 7137 tablet_bootstrap.cc:654] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:43.095357 7137 log.cc:824] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:43.097764 7138 tablet_bootstrap.cc:654] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:43.098188 7140 tablet_bootstrap.cc:492] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9: Bootstrap starting.
I20240501 08:10:43.098716 7137 tablet_bootstrap.cc:492] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa: No bootstrap required, opened a new log
I20240501 08:10:43.099009 7137 ts_tablet_manager.cc:1367] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20240501 08:10:43.099088 7138 log.cc:824] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:43.102185 7138 tablet_bootstrap.cc:492] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb: No bootstrap required, opened a new log
I20240501 08:10:43.102356 7140 tablet_bootstrap.cc:654] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:43.102488 7138 ts_tablet_manager.cc:1367] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 08:10:43.103643 7140 log.cc:824] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:43.105859 7137 raft_consensus.cc:348] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } }
I20240501 08:10:43.106231 7137 raft_consensus.cc:374] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:43.106343 7137 raft_consensus.cc:729] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aab9eecade541a285688996ff3e72aa, State: Initialized, Role: FOLLOWER
I20240501 08:10:43.106774 7140 tablet_bootstrap.cc:492] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9: No bootstrap required, opened a new log
I20240501 08:10:43.107136 7140 ts_tablet_manager.cc:1367] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20240501 08:10:43.107033 7137 consensus_queue.cc:260] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } }
I20240501 08:10:43.112752 7119 heartbeater.cc:503] Master 127.6.56.126:33021 was elected leader, sending a full tablet report...
I20240501 08:10:43.112558 7138 raft_consensus.cc:348] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } }
I20240501 08:10:43.113019 7138 raft_consensus.cc:374] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:43.113152 7138 raft_consensus.cc:729] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce6f663293db49408f32ff7a0c838bcb, State: Initialized, Role: FOLLOWER
I20240501 08:10:43.113427 7137 ts_tablet_manager.cc:1398] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20240501 08:10:43.113965 7138 consensus_queue.cc:260] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } }
I20240501 08:10:43.116602 7140 raft_consensus.cc:348] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } }
I20240501 08:10:43.117137 7140 raft_consensus.cc:374] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:43.117237 7138 ts_tablet_manager.cc:1398] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20240501 08:10:43.117244 7140 raft_consensus.cc:729] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5ea129cce047a3bc01be742085bbd9, State: Initialized, Role: FOLLOWER
I20240501 08:10:43.117966 7140 consensus_queue.cc:260] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } }
I20240501 08:10:43.119974 7140 ts_tablet_manager.cc:1398] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20240501 08:10:43.167359 7145 raft_consensus.cc:482] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:43.167675 7145 raft_consensus.cc:504] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } }
I20240501 08:10:43.169613 7145 leader_election.cc:288] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3aab9eecade541a285688996ff3e72aa (127.6.56.69:33851), ce6f663293db49408f32ff7a0c838bcb (127.6.56.65:43879)
I20240501 08:10:43.179145 6553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c944814a764167a88aa3b891fbb952" candidate_uuid: "0e5ea129cce047a3bc01be742085bbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6f663293db49408f32ff7a0c838bcb" is_pre_election: true
I20240501 08:10:43.179121 7074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c944814a764167a88aa3b891fbb952" candidate_uuid: "0e5ea129cce047a3bc01be742085bbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aab9eecade541a285688996ff3e72aa" is_pre_election: true
I20240501 08:10:43.179648 6553 raft_consensus.cc:2454] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e5ea129cce047a3bc01be742085bbd9 in term 0.
I20240501 08:10:43.179647 7074 raft_consensus.cc:2454] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e5ea129cce047a3bc01be742085bbd9 in term 0.
I20240501 08:10:43.180269 6618 leader_election.cc:302] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5ea129cce047a3bc01be742085bbd9, ce6f663293db49408f32ff7a0c838bcb; no voters:
I20240501 08:10:43.180691 7145 raft_consensus.cc:2781] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:43.180816 7145 raft_consensus.cc:482] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:43.180914 7145 raft_consensus.cc:3037] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:10:43.242548 7120 tablet.cc:2369] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:43.248237 7145 raft_consensus.cc:504] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } }
I20240501 08:10:43.249158 7145 leader_election.cc:288] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [CANDIDATE]: Term 1 election: Requested vote from peers 3aab9eecade541a285688996ff3e72aa (127.6.56.69:33851), ce6f663293db49408f32ff7a0c838bcb (127.6.56.65:43879)
I20240501 08:10:43.249660 7074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c944814a764167a88aa3b891fbb952" candidate_uuid: "0e5ea129cce047a3bc01be742085bbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3aab9eecade541a285688996ff3e72aa"
I20240501 08:10:43.249903 7074 raft_consensus.cc:3037] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:43.249913 6553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48c944814a764167a88aa3b891fbb952" candidate_uuid: "0e5ea129cce047a3bc01be742085bbd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce6f663293db49408f32ff7a0c838bcb"
I20240501 08:10:43.250116 6553 raft_consensus.cc:3037] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:43.252351 6553 raft_consensus.cc:2454] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5ea129cce047a3bc01be742085bbd9 in term 1.
I20240501 08:10:43.252838 6618 leader_election.cc:302] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5ea129cce047a3bc01be742085bbd9, ce6f663293db49408f32ff7a0c838bcb; no voters:
I20240501 08:10:43.253170 7145 raft_consensus.cc:2781] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:43.254113 7145 raft_consensus.cc:686] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [term 1 LEADER]: Becoming Leader. State: Replica: 0e5ea129cce047a3bc01be742085bbd9, State: Running, Role: LEADER
I20240501 08:10:43.254732 7145 consensus_queue.cc:237] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } }
I20240501 08:10:43.260825 6411 catalog_manager.cc:5484] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e5ea129cce047a3bc01be742085bbd9 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "0e5ea129cce047a3bc01be742085bbd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e5ea129cce047a3bc01be742085bbd9" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 42971 } health_report { overall_health: HEALTHY } } }
W20240501 08:10:43.278496 6730 tablet.cc:2369] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:10:43.301616 6599 tablet.cc:2369] T 48c944814a764167a88aa3b891fbb952 P ce6f663293db49408f32ff7a0c838bcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:43.303469 6369 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:10:43.306176 6369 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ce6f663293db49408f32ff7a0c838bcb to finish bootstrapping
I20240501 08:10:43.321751 6369 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0e5ea129cce047a3bc01be742085bbd9 to finish bootstrapping
I20240501 08:10:43.329708 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3774f5bbdb47493faeedddf0d02f647c to finish bootstrapping
I20240501 08:10:43.339079 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 3d65f90611c94a11bc46b045f9880f5f to finish bootstrapping
I20240501 08:10:43.345847 7074 raft_consensus.cc:2454] T 48c944814a764167a88aa3b891fbb952 P 3aab9eecade541a285688996ff3e72aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e5ea129cce047a3bc01be742085bbd9 in term 1.
I20240501 08:10:43.351549 6369 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3aab9eecade541a285688996ff3e72aa to finish bootstrapping
I20240501 08:10:43.518465 6989 heartbeater.cc:503] Master 127.6.56.126:33021 was elected leader, sending a full tablet report...
I20240501 08:10:43.777909 7165 consensus_queue.cc:1035] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce6f663293db49408f32ff7a0c838bcb" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 43879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:43.787569 7171 consensus_queue.cc:1035] T 48c944814a764167a88aa3b891fbb952 P 0e5ea129cce047a3bc01be742085bbd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aab9eecade541a285688996ff3e72aa" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:43.898864 6664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:43.899080 6794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:43.901880 6924 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:43.912915 7054 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:43.921352 6533 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:44.091384 6664 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:44.092213 6794 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:44.094759 6924 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:44.098002 7054 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:44.105032 6533 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:44.264531 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 6471
I20240501 08:10:44.265993 6592 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:44.361330 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 6471
I20240501 08:10:44.386484 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 6602
I20240501 08:10:44.387441 6723 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:44.481245 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 6602
W20240501 08:10:44.499722 7010 connection.cc:535] server connection from 127.6.56.66:41231 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:10:44.500305 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 6732
I20240501 08:10:44.501484 6853 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:44.589082 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 6732
I20240501 08:10:44.606393 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 6862
I20240501 08:10:44.607429 6984 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:44.694052 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 6862
I20240501 08:10:44.713102 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 6992
I20240501 08:10:44.715047 7113 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:44.808249 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 6992
W20240501 08:10:44.825402 6397 connection.cc:535] server connection from 127.6.56.69:38555 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:10:44.825703 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 6381
I20240501 08:10:44.826802 6442 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:44.934495 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 6381
2024-05-01T08:10:44Z chronyd exiting
[ OK ] AdminCliTest.RebalancerReportOnly (4989 ms)
[----------] 1 test from AdminCliTest (4989 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/0
I20240501 08:10:44.979143 6369 test_util.cc:268] Using random seed: 100415890
I20240501 08:10:44.982156 6369 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:10:44.982255 6369 ts_itest-base.cc:113] --------------
I20240501 08:10:44.982311 6369 ts_itest-base.cc:114] 5 tablet servers
I20240501 08:10:44.982388 6369 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:10:44.982445 6369 ts_itest-base.cc:116] --------------
2024-05-01T08:10:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:10:44Z Disabled control of system clock
I20240501 08:10:45.006282 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:43113
--webserver_interface=127.6.56.126
--webserver_port=0
--builtin_ntp_servers=127.6.56.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.56.126:43113
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 08:10:45.336023 7250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:45.336416 7250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:45.336561 7250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:45.344031 7250 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:45.344198 7250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:10:45.344281 7250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:45.344455 7250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:10:45.344522 7250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:10:45.353955 7250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.56.126:43113
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:43113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.56.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:45.355329 7250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:45.357074 7250 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:45.368223 7255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:45.369185 7258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:45.369885 7256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:45.370986 7250 server_base.cc:1034] running on GCE node
I20240501 08:10:45.371838 7250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:45.373426 7250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:45.374672 7250 hybrid_clock.cc:648] HybridClock initialized: now 1714551045374619 us; error 53 us; skew 500 ppm
I20240501 08:10:45.375044 7250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:45.377386 7250 webserver.cc:460] Webserver started at http://127.6.56.126:39015/ using document root <none> and password file <none>
I20240501 08:10:45.378728 7250 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:45.378857 7250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:45.379233 7250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:45.382047 7250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f3fb4eb2c36242e790ca14d019a4f7d3"
format_stamp: "Formatted at 2024-05-01 08:10:45 on dist-test-slave-f1nk"
I20240501 08:10:45.382846 7250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f3fb4eb2c36242e790ca14d019a4f7d3"
format_stamp: "Formatted at 2024-05-01 08:10:45 on dist-test-slave-f1nk"
I20240501 08:10:45.388551 7250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 08:10:45.392272 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:45.393772 7250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:45.393965 7250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f3fb4eb2c36242e790ca14d019a4f7d3"
format_stamp: "Formatted at 2024-05-01 08:10:45 on dist-test-slave-f1nk"
I20240501 08:10:45.394181 7250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:45.423328 7250 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:45.424619 7250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:45.424963 7250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:45.442806 7250 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.126:43113
I20240501 08:10:45.442844 7315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.126:43113 every 8 connection(s)
I20240501 08:10:45.444875 7250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:10:45.448614 7316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:45.453440 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 7250
I20240501 08:10:45.453640 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:10:45.458330 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3: Bootstrap starting.
I20240501 08:10:45.461808 7316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:45.463109 7316 log.cc:824] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:45.466809 7316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3: No bootstrap required, opened a new log
I20240501 08:10:45.473657 7316 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fb4eb2c36242e790ca14d019a4f7d3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 43113 } }
I20240501 08:10:45.474035 7316 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:45.474146 7316 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3fb4eb2c36242e790ca14d019a4f7d3, State: Initialized, Role: FOLLOWER
I20240501 08:10:45.474839 7316 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fb4eb2c36242e790ca14d019a4f7d3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 43113 } }
I20240501 08:10:45.475064 7316 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:45.475224 7316 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:45.475386 7316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:45.483006 7316 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fb4eb2c36242e790ca14d019a4f7d3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 43113 } }
I20240501 08:10:45.483626 7316 leader_election.cc:302] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3fb4eb2c36242e790ca14d019a4f7d3; no voters:
I20240501 08:10:45.484447 7316 leader_election.cc:288] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:45.484632 7321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:45.485806 7321 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [term 1 LEADER]: Becoming Leader. State: Replica: f3fb4eb2c36242e790ca14d019a4f7d3, State: Running, Role: LEADER
I20240501 08:10:45.487046 7316 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:45.486894 7321 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fb4eb2c36242e790ca14d019a4f7d3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 43113 } }
I20240501 08:10:45.490272 7322 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3fb4eb2c36242e790ca14d019a4f7d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fb4eb2c36242e790ca14d019a4f7d3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 43113 } } }
I20240501 08:10:45.490721 7322 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:45.491386 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3fb4eb2c36242e790ca14d019a4f7d3. Latest consensus state: current_term: 1 leader_uuid: "f3fb4eb2c36242e790ca14d019a4f7d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3fb4eb2c36242e790ca14d019a4f7d3" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 43113 } } }
I20240501 08:10:45.491745 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3 [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:45.492367 7330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:45.497749 7330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:45.507112 7330 catalog_manager.cc:1347] Generated new cluster ID: 62ff7bcb7f4c4d97976c6d57212d7973
I20240501 08:10:45.507258 7330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:45.517855 7330 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:45.519059 7330 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:45.536921 7330 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f3fb4eb2c36242e790ca14d019a4f7d3: Generated new TSK 0
I20240501 08:10:45.537847 7330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:45.543553 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.65:0
--local_ip_for_outbound_sockets=127.6.56.65
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--builtin_ntp_servers=127.6.56.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:10:45.900113 7340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:45.900472 7340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:45.900617 7340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:45.908226 7340 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:45.908401 7340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:45.908587 7340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.65
I20240501 08:10:45.917811 7340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.65
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:45.919217 7340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:45.921054 7340 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:45.933486 7348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:45.934386 7346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:45.935190 7345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:45.936105 7340 server_base.cc:1034] running on GCE node
I20240501 08:10:45.936795 7340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:45.937789 7340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:45.938988 7340 hybrid_clock.cc:648] HybridClock initialized: now 1714551045938938 us; error 55 us; skew 500 ppm
I20240501 08:10:45.939296 7340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:45.941659 7340 webserver.cc:460] Webserver started at http://127.6.56.65:44547/ using document root <none> and password file <none>
I20240501 08:10:45.942426 7340 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:45.942543 7340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:45.942867 7340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:45.945613 7340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "85cf24337aa647e7baa67af70220feac"
format_stamp: "Formatted at 2024-05-01 08:10:45 on dist-test-slave-f1nk"
I20240501 08:10:45.946353 7340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "85cf24337aa647e7baa67af70220feac"
format_stamp: "Formatted at 2024-05-01 08:10:45 on dist-test-slave-f1nk"
I20240501 08:10:45.951495 7340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:10:45.954962 7354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:45.956357 7340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:45.956523 7340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "85cf24337aa647e7baa67af70220feac"
format_stamp: "Formatted at 2024-05-01 08:10:45 on dist-test-slave-f1nk"
I20240501 08:10:45.956720 7340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:45.991211 7340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:45.992097 7340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:45.992424 7340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:45.993427 7340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:45.995007 7340 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:45.995126 7340 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:45.995214 7340 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:45.995268 7340 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:46.069742 7340 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.65:38165
I20240501 08:10:46.069815 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.65:38165 every 8 connection(s)
I20240501 08:10:46.071151 7340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:10:46.076740 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 7340
I20240501 08:10:46.076905 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:10:46.080080 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.66:0
--local_ip_for_outbound_sockets=127.6.56.66
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--builtin_ntp_servers=127.6.56.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:46.084285 7467 heartbeater.cc:347] Connected to a master server at 127.6.56.126:43113
I20240501 08:10:46.084627 7467 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:46.085417 7467 heartbeater.cc:511] Master 127.6.56.126:43113 requested a full tablet report, sending...
I20240501 08:10:46.087476 7281 ts_manager.cc:194] Registered new tserver with Master: 85cf24337aa647e7baa67af70220feac (127.6.56.65:38165)
I20240501 08:10:46.089210 7281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.65:53043
W20240501 08:10:46.426954 7471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:46.427325 7471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:46.427451 7471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:46.435117 7471 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:46.435258 7471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:46.435420 7471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.66
I20240501 08:10:46.444885 7471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.66
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:46.446208 7471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:46.448041 7471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:46.461130 7476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:46.462517 7477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:46.462821 7479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:46.463722 7471 server_base.cc:1034] running on GCE node
I20240501 08:10:46.464390 7471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:46.465404 7471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:46.466612 7471 hybrid_clock.cc:648] HybridClock initialized: now 1714551046466562 us; error 49 us; skew 500 ppm
I20240501 08:10:46.466945 7471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:46.471246 7471 webserver.cc:460] Webserver started at http://127.6.56.66:35331/ using document root <none> and password file <none>
I20240501 08:10:46.472033 7471 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:46.472164 7471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:46.472556 7471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:46.475417 7471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "90e7d32f392b4394972d6d2240d17108"
format_stamp: "Formatted at 2024-05-01 08:10:46 on dist-test-slave-f1nk"
I20240501 08:10:46.476212 7471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "90e7d32f392b4394972d6d2240d17108"
format_stamp: "Formatted at 2024-05-01 08:10:46 on dist-test-slave-f1nk"
I20240501 08:10:46.481735 7471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 08:10:46.485132 7485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:46.486460 7471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:46.486616 7471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "90e7d32f392b4394972d6d2240d17108"
format_stamp: "Formatted at 2024-05-01 08:10:46 on dist-test-slave-f1nk"
I20240501 08:10:46.486809 7471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:46.504176 7471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:46.505095 7471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:46.505388 7471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:46.506346 7471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:46.507866 7471 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:46.507973 7471 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:46.508091 7471 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:46.508172 7471 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:46.539939 7471 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.66:33023
I20240501 08:10:46.539985 7597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.66:33023 every 8 connection(s)
I20240501 08:10:46.541332 7471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:10:46.549640 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 7471
I20240501 08:10:46.549844 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:10:46.553309 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.67:0
--local_ip_for_outbound_sockets=127.6.56.67
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--builtin_ntp_servers=127.6.56.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:46.554071 7598 heartbeater.cc:347] Connected to a master server at 127.6.56.126:43113
I20240501 08:10:46.554366 7598 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:46.555099 7598 heartbeater.cc:511] Master 127.6.56.126:43113 requested a full tablet report, sending...
I20240501 08:10:46.556560 7281 ts_manager.cc:194] Registered new tserver with Master: 90e7d32f392b4394972d6d2240d17108 (127.6.56.66:33023)
I20240501 08:10:46.557399 7281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.66:36327
W20240501 08:10:46.895223 7601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:46.895570 7601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:46.895722 7601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:46.903407 7601 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:46.903568 7601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:46.903744 7601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.67
I20240501 08:10:46.912842 7601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.67
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:46.914203 7601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:46.915926 7601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:46.928573 7606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:46.929042 7609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:46.929970 7607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:46.930778 7601 server_base.cc:1034] running on GCE node
I20240501 08:10:46.931438 7601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:46.932485 7601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:46.933727 7601 hybrid_clock.cc:648] HybridClock initialized: now 1714551046933672 us; error 48 us; skew 500 ppm
I20240501 08:10:46.934120 7601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:46.936571 7601 webserver.cc:460] Webserver started at http://127.6.56.67:43699/ using document root <none> and password file <none>
I20240501 08:10:46.937346 7601 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:46.937465 7601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:46.937796 7601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:46.940649 7601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fb0964271ec54a33ae61371961eb3864"
format_stamp: "Formatted at 2024-05-01 08:10:46 on dist-test-slave-f1nk"
I20240501 08:10:46.941452 7601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fb0964271ec54a33ae61371961eb3864"
format_stamp: "Formatted at 2024-05-01 08:10:46 on dist-test-slave-f1nk"
I20240501 08:10:46.947053 7601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:10:46.950632 7615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:46.951968 7601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:10:46.952127 7601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fb0964271ec54a33ae61371961eb3864"
format_stamp: "Formatted at 2024-05-01 08:10:46 on dist-test-slave-f1nk"
I20240501 08:10:46.952347 7601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:46.971190 7601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:46.972079 7601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:46.972404 7601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:46.973457 7601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:46.975085 7601 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:46.975183 7601 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:46.975302 7601 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:46.975379 7601 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.009794 7601 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.67:36445
I20240501 08:10:47.009846 7727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.67:36445 every 8 connection(s)
I20240501 08:10:47.011121 7601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:10:47.011586 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 7601
I20240501 08:10:47.011756 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:10:47.015441 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.68:0
--local_ip_for_outbound_sockets=127.6.56.68
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--builtin_ntp_servers=127.6.56.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:47.026201 7728 heartbeater.cc:347] Connected to a master server at 127.6.56.126:43113
I20240501 08:10:47.026540 7728 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.027309 7728 heartbeater.cc:511] Master 127.6.56.126:43113 requested a full tablet report, sending...
I20240501 08:10:47.028798 7281 ts_manager.cc:194] Registered new tserver with Master: fb0964271ec54a33ae61371961eb3864 (127.6.56.67:36445)
I20240501 08:10:47.029559 7281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.67:49137
I20240501 08:10:47.092197 7467 heartbeater.cc:503] Master 127.6.56.126:43113 was elected leader, sending a full tablet report...
W20240501 08:10:47.350715 7731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:47.351023 7731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:47.351120 7731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:47.358645 7731 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:47.358783 7731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:47.358906 7731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.68
I20240501 08:10:47.368371 7731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.68
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:47.369666 7731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:47.371299 7731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 08:10:47.383903 7731 server_base.cc:1034] running on GCE node
W20240501 08:10:47.384747 7737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.385447 7739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.385525 7736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.386230 7731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:47.387132 7731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:47.388355 7731 hybrid_clock.cc:648] HybridClock initialized: now 1714551047388304 us; error 36 us; skew 500 ppm
I20240501 08:10:47.388670 7731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:47.391047 7731 webserver.cc:460] Webserver started at http://127.6.56.68:35003/ using document root <none> and password file <none>
I20240501 08:10:47.391851 7731 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:47.391965 7731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:47.392333 7731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:47.394982 7731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b8058022323e49cc90a99eb9a3e3f049"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-f1nk"
I20240501 08:10:47.395754 7731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b8058022323e49cc90a99eb9a3e3f049"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-f1nk"
I20240501 08:10:47.401060 7731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20240501 08:10:47.404430 7745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.405715 7731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 08:10:47.405871 7731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b8058022323e49cc90a99eb9a3e3f049"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-f1nk"
I20240501 08:10:47.406070 7731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:47.424894 7731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:47.425721 7731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:47.425982 7731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:47.426903 7731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:47.428398 7731 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:47.428485 7731 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.428607 7731 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:47.428682 7731 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.462076 7731 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.68:37945
I20240501 08:10:47.462139 7857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.68:37945 every 8 connection(s)
I20240501 08:10:47.463411 7731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:10:47.464542 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 7731
I20240501 08:10:47.464748 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:10:47.468511 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.69:0
--local_ip_for_outbound_sockets=127.6.56.69
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--builtin_ntp_servers=127.6.56.84:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:47.478368 7858 heartbeater.cc:347] Connected to a master server at 127.6.56.126:43113
I20240501 08:10:47.478682 7858 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.479470 7858 heartbeater.cc:511] Master 127.6.56.126:43113 requested a full tablet report, sending...
I20240501 08:10:47.481030 7281 ts_manager.cc:194] Registered new tserver with Master: b8058022323e49cc90a99eb9a3e3f049 (127.6.56.68:37945)
I20240501 08:10:47.481832 7281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.68:40183
I20240501 08:10:47.559942 7598 heartbeater.cc:503] Master 127.6.56.126:43113 was elected leader, sending a full tablet report...
W20240501 08:10:47.801824 7861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:47.802176 7861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:47.802326 7861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:47.810048 7861 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:47.810220 7861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:47.810397 7861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.69
I20240501 08:10:47.819691 7861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:43113
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.69
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:47.821120 7861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:47.822932 7861 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:47.835917 7866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.836856 7867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:47.837527 7869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:47.838445 7861 server_base.cc:1034] running on GCE node
I20240501 08:10:47.839087 7861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:47.840122 7861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:47.841331 7861 hybrid_clock.cc:648] HybridClock initialized: now 1714551047841259 us; error 67 us; skew 500 ppm
I20240501 08:10:47.841655 7861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:47.843945 7861 webserver.cc:460] Webserver started at http://127.6.56.69:36067/ using document root <none> and password file <none>
I20240501 08:10:47.844746 7861 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:47.844866 7861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:47.845186 7861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:47.847772 7861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "704de15ed69a4ddab965eec0a04e24b7"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-f1nk"
I20240501 08:10:47.848609 7861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "704de15ed69a4ddab965eec0a04e24b7"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-f1nk"
I20240501 08:10:47.853675 7861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240501 08:10:47.857230 7875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.858768 7861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 08:10:47.858935 7861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "704de15ed69a4ddab965eec0a04e24b7"
format_stamp: "Formatted at 2024-05-01 08:10:47 on dist-test-slave-f1nk"
I20240501 08:10:47.859139 7861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:47.873451 7861 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:47.874297 7861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:47.874567 7861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:47.875540 7861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:47.877049 7861 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:47.877138 7861 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.877223 7861 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:47.877275 7861 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:47.909572 7861 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.69:38133
I20240501 08:10:47.909619 7987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.69:38133 every 8 connection(s)
I20240501 08:10:47.911006 7861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:10:47.918560 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 7861
I20240501 08:10:47.918753 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceStartCriteriaTest.TabletServerIsDown_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:10:47.923614 7988 heartbeater.cc:347] Connected to a master server at 127.6.56.126:43113
I20240501 08:10:47.923878 7988 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:47.924625 7988 heartbeater.cc:511] Master 127.6.56.126:43113 requested a full tablet report, sending...
I20240501 08:10:47.925946 7281 ts_manager.cc:194] Registered new tserver with Master: 704de15ed69a4ddab965eec0a04e24b7 (127.6.56.69:38133)
I20240501 08:10:47.926693 7281 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.69:33977
I20240501 08:10:47.934574 6369 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:10:47.951192 7281 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46982:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 08:10:47.983969 7533 tablet_service.cc:1467] Processing CreateTablet for tablet 9a727183d9084e2ab36e9ffa209c7704 (DEFAULT_TABLE table=TestTable [id=4cd68af8c3a54cae8d5cbe612d55d349]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.984601 7793 tablet_service.cc:1467] Processing CreateTablet for tablet 9a727183d9084e2ab36e9ffa209c7704 (DEFAULT_TABLE table=TestTable [id=4cd68af8c3a54cae8d5cbe612d55d349]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.984956 7923 tablet_service.cc:1467] Processing CreateTablet for tablet 9a727183d9084e2ab36e9ffa209c7704 (DEFAULT_TABLE table=TestTable [id=4cd68af8c3a54cae8d5cbe612d55d349]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:47.985636 7533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a727183d9084e2ab36e9ffa209c7704. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.985772 7793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a727183d9084e2ab36e9ffa209c7704. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.986331 7923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a727183d9084e2ab36e9ffa209c7704. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:47.996407 8006 tablet_bootstrap.cc:492] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049: Bootstrap starting.
I20240501 08:10:47.999169 8007 tablet_bootstrap.cc:492] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108: Bootstrap starting.
I20240501 08:10:48.000662 8008 tablet_bootstrap.cc:492] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7: Bootstrap starting.
I20240501 08:10:48.002408 8006 tablet_bootstrap.cc:654] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.002792 8007 tablet_bootstrap.cc:654] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.003587 8008 tablet_bootstrap.cc:654] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:48.003757 8006 log.cc:824] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:48.003899 8007 log.cc:824] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:48.004729 8008 log.cc:824] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:48.007210 8008 tablet_bootstrap.cc:492] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7: No bootstrap required, opened a new log
I20240501 08:10:48.007292 8006 tablet_bootstrap.cc:492] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049: No bootstrap required, opened a new log
I20240501 08:10:48.007290 8007 tablet_bootstrap.cc:492] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108: No bootstrap required, opened a new log
I20240501 08:10:48.007486 8008 ts_tablet_manager.cc:1367] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:10:48.007580 8007 ts_tablet_manager.cc:1367] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 08:10:48.007578 8006 ts_tablet_manager.cc:1367] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 08:10:48.016616 8007 raft_consensus.cc:348] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.016884 8006 raft_consensus.cc:348] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.017323 8006 raft_consensus.cc:374] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.017300 8007 raft_consensus.cc:374] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.017429 8007 raft_consensus.cc:729] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e7d32f392b4394972d6d2240d17108, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.017455 8006 raft_consensus.cc:729] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8058022323e49cc90a99eb9a3e3f049, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.017243 8008 raft_consensus.cc:348] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.017648 8008 raft_consensus.cc:374] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:48.017757 8008 raft_consensus.cc:729] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 704de15ed69a4ddab965eec0a04e24b7, State: Initialized, Role: FOLLOWER
I20240501 08:10:48.018182 8006 consensus_queue.cc:260] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.018182 8007 consensus_queue.cc:260] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.018353 8008 consensus_queue.cc:260] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.020139 8006 ts_tablet_manager.cc:1398] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20240501 08:10:48.020517 8007 ts_tablet_manager.cc:1398] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
I20240501 08:10:48.021710 7858 heartbeater.cc:503] Master 127.6.56.126:43113 was elected leader, sending a full tablet report...
I20240501 08:10:48.025117 7988 heartbeater.cc:503] Master 127.6.56.126:43113 was elected leader, sending a full tablet report...
I20240501 08:10:48.025822 8008 ts_tablet_manager.cc:1398] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7: Time spent starting tablet: real 0.018s user 0.016s sys 0.000s
I20240501 08:10:48.031929 7728 heartbeater.cc:503] Master 127.6.56.126:43113 was elected leader, sending a full tablet report...
W20240501 08:10:48.043920 7599 tablet.cc:2369] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:10:48.162770 7989 tablet.cc:2369] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:10:48.215505 7859 tablet.cc:2369] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:10:48.230893 8013 raft_consensus.cc:482] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:48.231238 8013 raft_consensus.cc:504] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.233186 8013 leader_election.cc:288] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90e7d32f392b4394972d6d2240d17108 (127.6.56.66:33023), 704de15ed69a4ddab965eec0a04e24b7 (127.6.56.69:38133)
I20240501 08:10:48.234094 8014 raft_consensus.cc:482] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:10:48.234508 8014 raft_consensus.cc:504] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.237645 8014 leader_election.cc:288] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90e7d32f392b4394972d6d2240d17108 (127.6.56.66:33023), b8058022323e49cc90a99eb9a3e3f049 (127.6.56.68:37945)
I20240501 08:10:48.245831 7943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a727183d9084e2ab36e9ffa209c7704" candidate_uuid: "b8058022323e49cc90a99eb9a3e3f049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704de15ed69a4ddab965eec0a04e24b7" is_pre_election: true
I20240501 08:10:48.246399 7943 raft_consensus.cc:2454] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8058022323e49cc90a99eb9a3e3f049 in term 0.
I20240501 08:10:48.246769 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a727183d9084e2ab36e9ffa209c7704" candidate_uuid: "b8058022323e49cc90a99eb9a3e3f049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e7d32f392b4394972d6d2240d17108" is_pre_election: true
I20240501 08:10:48.247233 7746 leader_election.cc:302] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 704de15ed69a4ddab965eec0a04e24b7, b8058022323e49cc90a99eb9a3e3f049; no voters:
I20240501 08:10:48.247332 7553 raft_consensus.cc:2454] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8058022323e49cc90a99eb9a3e3f049 in term 0.
I20240501 08:10:48.247738 8013 raft_consensus.cc:2781] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:48.247962 8013 raft_consensus.cc:482] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:48.248143 8013 raft_consensus.cc:3037] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.250746 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a727183d9084e2ab36e9ffa209c7704" candidate_uuid: "704de15ed69a4ddab965eec0a04e24b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e7d32f392b4394972d6d2240d17108" is_pre_election: true
I20240501 08:10:48.250979 7553 raft_consensus.cc:2454] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 704de15ed69a4ddab965eec0a04e24b7 in term 0.
I20240501 08:10:48.251471 7879 leader_election.cc:302] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 704de15ed69a4ddab965eec0a04e24b7, 90e7d32f392b4394972d6d2240d17108; no voters:
I20240501 08:10:48.251909 8014 raft_consensus.cc:2781] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:10:48.251535 8013 raft_consensus.cc:504] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.252058 8014 raft_consensus.cc:482] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:10:48.252194 8014 raft_consensus.cc:3037] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.252512 7813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a727183d9084e2ab36e9ffa209c7704" candidate_uuid: "704de15ed69a4ddab965eec0a04e24b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8058022323e49cc90a99eb9a3e3f049" is_pre_election: true
I20240501 08:10:48.253041 7813 raft_consensus.cc:2379] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 704de15ed69a4ddab965eec0a04e24b7 in current term 1: Already voted for candidate b8058022323e49cc90a99eb9a3e3f049 in this term.
I20240501 08:10:48.253036 7553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a727183d9084e2ab36e9ffa209c7704" candidate_uuid: "b8058022323e49cc90a99eb9a3e3f049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e7d32f392b4394972d6d2240d17108"
I20240501 08:10:48.253098 7943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a727183d9084e2ab36e9ffa209c7704" candidate_uuid: "b8058022323e49cc90a99eb9a3e3f049" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "704de15ed69a4ddab965eec0a04e24b7"
I20240501 08:10:48.253240 7553 raft_consensus.cc:3037] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:48.253845 8013 leader_election.cc:288] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [CANDIDATE]: Term 1 election: Requested vote from peers 90e7d32f392b4394972d6d2240d17108 (127.6.56.66:33023), 704de15ed69a4ddab965eec0a04e24b7 (127.6.56.69:38133)
I20240501 08:10:48.255084 8014 raft_consensus.cc:504] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.255563 7553 raft_consensus.cc:2454] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8058022323e49cc90a99eb9a3e3f049 in term 1.
I20240501 08:10:48.255685 8014 leader_election.cc:288] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [CANDIDATE]: Term 1 election: Requested vote from peers 90e7d32f392b4394972d6d2240d17108 (127.6.56.66:33023), b8058022323e49cc90a99eb9a3e3f049 (127.6.56.68:37945)
I20240501 08:10:48.256008 7943 raft_consensus.cc:2379] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b8058022323e49cc90a99eb9a3e3f049 in current term 1: Already voted for candidate 704de15ed69a4ddab965eec0a04e24b7 in this term.
I20240501 08:10:48.256008 7749 leader_election.cc:302] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90e7d32f392b4394972d6d2240d17108, b8058022323e49cc90a99eb9a3e3f049; no voters:
I20240501 08:10:48.256196 7813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a727183d9084e2ab36e9ffa209c7704" candidate_uuid: "704de15ed69a4ddab965eec0a04e24b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8058022323e49cc90a99eb9a3e3f049"
I20240501 08:10:48.256455 7813 raft_consensus.cc:2379] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 704de15ed69a4ddab965eec0a04e24b7 in current term 1: Already voted for candidate b8058022323e49cc90a99eb9a3e3f049 in this term.
I20240501 08:10:48.256419 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a727183d9084e2ab36e9ffa209c7704" candidate_uuid: "704de15ed69a4ddab965eec0a04e24b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e7d32f392b4394972d6d2240d17108"
I20240501 08:10:48.256599 8013 raft_consensus.cc:2781] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:48.256773 7552 raft_consensus.cc:2379] T 9a727183d9084e2ab36e9ffa209c7704 P 90e7d32f392b4394972d6d2240d17108 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 704de15ed69a4ddab965eec0a04e24b7 in current term 1: Already voted for candidate b8058022323e49cc90a99eb9a3e3f049 in this term.
I20240501 08:10:48.257884 7879 leader_election.cc:302] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 704de15ed69a4ddab965eec0a04e24b7; no voters: 90e7d32f392b4394972d6d2240d17108, b8058022323e49cc90a99eb9a3e3f049
I20240501 08:10:48.257926 8013 raft_consensus.cc:686] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [term 1 LEADER]: Becoming Leader. State: Replica: b8058022323e49cc90a99eb9a3e3f049, State: Running, Role: LEADER
I20240501 08:10:48.258212 8014 raft_consensus.cc:2726] T 9a727183d9084e2ab36e9ffa209c7704 P 704de15ed69a4ddab965eec0a04e24b7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:10:48.258479 8013 consensus_queue.cc:237] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } }
I20240501 08:10:48.262966 7279 catalog_manager.cc:5484] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8058022323e49cc90a99eb9a3e3f049 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "b8058022323e49cc90a99eb9a3e3f049" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 } } peers { permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 } } peers { permanent_uuid: "b8058022323e49cc90a99eb9a3e3f049" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 37945 } } }
I20240501 08:10:48.342864 6369 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 08:10:48.344918 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 85cf24337aa647e7baa67af70220feac to finish bootstrapping
I20240501 08:10:48.353817 6369 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 90e7d32f392b4394972d6d2240d17108 to finish bootstrapping
I20240501 08:10:48.361536 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fb0964271ec54a33ae61371961eb3864 to finish bootstrapping
I20240501 08:10:48.370471 6369 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b8058022323e49cc90a99eb9a3e3f049 to finish bootstrapping
I20240501 08:10:48.377751 6369 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 704de15ed69a4ddab965eec0a04e24b7 to finish bootstrapping
I20240501 08:10:48.385187 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 7340
I20240501 08:10:48.386225 7462 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:48.468312 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 7340
I20240501 08:10:48.757510 8013 consensus_queue.cc:1035] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e7d32f392b4394972d6d2240d17108" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 33023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.767309 8028 consensus_queue.cc:1035] T 9a727183d9084e2ab36e9ffa209c7704 P b8058022323e49cc90a99eb9a3e3f049 [LEADER]: Connected to new peer: Peer: permanent_uuid: "704de15ed69a4ddab965eec0a04e24b7" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:10:48.956529 7533 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.965003 7923 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:48.970889 7793 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:48.979061 7663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.106655 7923 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.116267 7533 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.119962 7663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.126188 7793 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:49.284561 7533 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.285436 7923 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.292268 7663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.334443 7793 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:49.949863 7923 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.951972 7533 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:49.956820 7793 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:49.958868 7663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:50.088824 7923 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:50.092235 7533 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:50.097299 7793 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:50.099736 7663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:50.271231 7533 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:50.272557 7923 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:50.275401 7793 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:10:50.287621 7663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:10:50.432202 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 7471
I20240501 08:10:50.434607 7593 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:50.532888 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 7471
I20240501 08:10:50.551625 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 7601
I20240501 08:10:50.552718 7723 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:50.645475 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 7601
I20240501 08:10:50.662865 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 7731
I20240501 08:10:50.663934 7852 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:50.764010 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 7731
I20240501 08:10:50.782470 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 7861
I20240501 08:10:50.783488 7983 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:50.881621 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 7861
W20240501 08:10:50.901786 7266 connection.cc:535] server connection from 127.6.56.69:33977 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:10:50.902088 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 7250
I20240501 08:10:50.903232 7311 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:51.021536 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 7250
2024-05-01T08:10:51Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/0 (6085 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6085 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/0
I20240501 08:10:51.064762 6369 test_util.cc:268] Using random seed: 106501537
I20240501 08:10:51.066673 6369 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:10:51.066743 6369 ts_itest-base.cc:113] --------------
I20240501 08:10:51.066794 6369 ts_itest-base.cc:114] 7 tablet servers
I20240501 08:10:51.066848 6369 ts_itest-base.cc:115] 1 replicas per TS
I20240501 08:10:51.066912 6369 ts_itest-base.cc:116] --------------
2024-05-01T08:10:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:10:51Z Disabled control of system clock
I20240501 08:10:51.090934 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:36943
--webserver_interface=127.6.56.126
--webserver_port=0
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.56.126:36943
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 08:10:51.420486 8232 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 08:10:51.420812 8232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:51.420914 8232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:51.421010 8232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:51.428503 8232 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:51.428624 8232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:10:51.428704 8232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:51.428764 8232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:10:51.428853 8232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:10:51.438030 8232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.56.126:36943
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:36943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.56.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:51.439394 8232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:51.441200 8232 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:51.452154 8237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:51.453092 8238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:51.456418 8240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:51.456472 8232 server_base.cc:1034] running on GCE node
I20240501 08:10:51.457388 8232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:51.458846 8232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:51.460078 8232 hybrid_clock.cc:648] HybridClock initialized: now 1714551051460018 us; error 38 us; skew 500 ppm
I20240501 08:10:51.460466 8232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:51.466816 8232 webserver.cc:460] Webserver started at http://127.6.56.126:33287/ using document root <none> and password file <none>
I20240501 08:10:51.467676 8232 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:51.467825 8232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:51.468168 8232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:51.470937 8232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0276005c13714ca09390ff319a3ff96a"
format_stamp: "Formatted at 2024-05-01 08:10:51 on dist-test-slave-f1nk"
I20240501 08:10:51.471720 8232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0276005c13714ca09390ff319a3ff96a"
format_stamp: "Formatted at 2024-05-01 08:10:51 on dist-test-slave-f1nk"
I20240501 08:10:51.477015 8232 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240501 08:10:51.480490 8246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:51.481879 8232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240501 08:10:51.482060 8232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0276005c13714ca09390ff319a3ff96a"
format_stamp: "Formatted at 2024-05-01 08:10:51 on dist-test-slave-f1nk"
I20240501 08:10:51.482261 8232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:51.511689 8232 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:51.512668 8232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:51.512974 8232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:51.531229 8232 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.126:36943
I20240501 08:10:51.531247 8297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.126:36943 every 8 connection(s)
I20240501 08:10:51.532694 8232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:10:51.536661 8298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:51.538933 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 8232
I20240501 08:10:51.539134 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:10:51.547709 8298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a: Bootstrap starting.
I20240501 08:10:51.551256 8298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:51.552330 8298 log.cc:824] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:51.554945 8298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a: No bootstrap required, opened a new log
I20240501 08:10:51.561465 8298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276005c13714ca09390ff319a3ff96a" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36943 } }
I20240501 08:10:51.561841 8298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:51.561941 8298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0276005c13714ca09390ff319a3ff96a, State: Initialized, Role: FOLLOWER
I20240501 08:10:51.562558 8298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276005c13714ca09390ff319a3ff96a" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36943 } }
I20240501 08:10:51.562762 8298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:51.562906 8298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:51.563050 8298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:51.565238 8298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276005c13714ca09390ff319a3ff96a" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36943 } }
I20240501 08:10:51.565775 8298 leader_election.cc:302] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0276005c13714ca09390ff319a3ff96a; no voters:
I20240501 08:10:51.566383 8298 leader_election.cc:288] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:51.566536 8303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:51.567950 8303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [term 1 LEADER]: Becoming Leader. State: Replica: 0276005c13714ca09390ff319a3ff96a, State: Running, Role: LEADER
I20240501 08:10:51.568511 8303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276005c13714ca09390ff319a3ff96a" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36943 } }
I20240501 08:10:51.568642 8298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:10:51.571745 8305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0276005c13714ca09390ff319a3ff96a. Latest consensus state: current_term: 1 leader_uuid: "0276005c13714ca09390ff319a3ff96a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276005c13714ca09390ff319a3ff96a" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36943 } } }
I20240501 08:10:51.572027 8305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:51.572477 8304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0276005c13714ca09390ff319a3ff96a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0276005c13714ca09390ff319a3ff96a" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36943 } } }
I20240501 08:10:51.572753 8304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a [sys.catalog]: This master's current role is: LEADER
I20240501 08:10:51.572897 8312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:10:51.578619 8312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:10:51.587647 8312 catalog_manager.cc:1347] Generated new cluster ID: dc6cf326ff3d4071b4a0899a4276417c
I20240501 08:10:51.587800 8312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:10:51.607522 8312 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:10:51.609422 8312 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:10:51.627454 8312 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0276005c13714ca09390ff319a3ff96a: Generated new TSK 0
I20240501 08:10:51.628582 8312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:10:51.646473 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.65:0
--local_ip_for_outbound_sockets=127.6.56.65
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:10:51.979192 8322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:51.979503 8322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:51.979589 8322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:51.987560 8322 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:51.987695 8322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:51.987836 8322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.65
I20240501 08:10:51.996999 8322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.65
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:51.998320 8322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:52.000021 8322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:52.012573 8330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:52.013165 8328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:52.014081 8327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:52.014922 8322 server_base.cc:1034] running on GCE node
I20240501 08:10:52.015518 8322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:52.016525 8322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:52.017741 8322 hybrid_clock.cc:648] HybridClock initialized: now 1714551052017684 us; error 59 us; skew 500 ppm
I20240501 08:10:52.018056 8322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:52.020422 8322 webserver.cc:460] Webserver started at http://127.6.56.65:46323/ using document root <none> and password file <none>
I20240501 08:10:52.021183 8322 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:52.021287 8322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:52.021591 8322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:52.024235 8322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:10:52.025097 8322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:10:52.030244 8322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:10:52.033793 8336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.035174 8322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:10:52.035334 8322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:10:52.035535 8322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:52.062992 8322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:52.063916 8322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:52.064244 8322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:52.065271 8322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:52.066743 8322 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:52.066828 8322 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.066943 8322 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:52.067023 8322 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.099911 8322 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.65:46073
I20240501 08:10:52.099969 8448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.65:46073 every 8 connection(s)
I20240501 08:10:52.101298 8322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:10:52.105249 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 8322
I20240501 08:10:52.105499 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:10:52.109069 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.66:0
--local_ip_for_outbound_sockets=127.6.56.66
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:52.115278 8449 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:10:52.115587 8449 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:52.116338 8449 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:10:52.118465 8263 ts_manager.cc:194] Registered new tserver with Master: fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:10:52.120473 8263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.65:37511
W20240501 08:10:52.442301 8453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:52.442638 8453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:52.442777 8453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:52.450549 8453 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:52.450695 8453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:52.450855 8453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.66
I20240501 08:10:52.459941 8453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.66
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:52.461336 8453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:52.463035 8453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:52.475983 8459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:52.476752 8461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:52.477469 8458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:52.478474 8453 server_base.cc:1034] running on GCE node
I20240501 08:10:52.479097 8453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:52.480096 8453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:52.481287 8453 hybrid_clock.cc:648] HybridClock initialized: now 1714551052481227 us; error 63 us; skew 500 ppm
I20240501 08:10:52.481603 8453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:52.483902 8453 webserver.cc:460] Webserver started at http://127.6.56.66:44589/ using document root <none> and password file <none>
I20240501 08:10:52.484714 8453 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:52.484812 8453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:52.485082 8453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:52.487861 8453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "2631692a8bb44abfb7d05022e6ae9b94"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:10:52.488659 8453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "2631692a8bb44abfb7d05022e6ae9b94"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:10:52.493849 8453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240501 08:10:52.497463 8467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.498901 8453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 08:10:52.499049 8453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "2631692a8bb44abfb7d05022e6ae9b94"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:10:52.499245 8453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:52.516150 8453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:52.517051 8453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:52.517387 8453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:52.518474 8453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:52.519960 8453 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:52.520046 8453 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.520134 8453 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:52.520191 8453 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.554745 8453 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.66:40791
I20240501 08:10:52.554800 8579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.66:40791 every 8 connection(s)
I20240501 08:10:52.556080 8453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:10:52.558854 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 8453
I20240501 08:10:52.559016 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:10:52.561976 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.67:0
--local_ip_for_outbound_sockets=127.6.56.67
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:52.569555 8580 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:10:52.569911 8580 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:52.570833 8580 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:10:52.572453 8263 ts_manager.cc:194] Registered new tserver with Master: 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:10:52.573244 8263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.66:40681
W20240501 08:10:52.903353 8583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:52.903667 8583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:52.903805 8583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:52.911617 8583 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:52.911800 8583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:52.911972 8583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.67
I20240501 08:10:52.921447 8583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.67
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:52.922899 8583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:52.924815 8583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:52.938007 8588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:52.938735 8589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:52.940577 8591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:52.940546 8583 server_base.cc:1034] running on GCE node
I20240501 08:10:52.941320 8583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:52.942315 8583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:52.943518 8583 hybrid_clock.cc:648] HybridClock initialized: now 1714551052943462 us; error 60 us; skew 500 ppm
I20240501 08:10:52.943826 8583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:52.946238 8583 webserver.cc:460] Webserver started at http://127.6.56.67:33321/ using document root <none> and password file <none>
I20240501 08:10:52.946976 8583 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:52.947093 8583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:52.947414 8583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:52.950148 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0ac549325eca4b79be5c3861c96f23b2"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:10:52.950986 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0ac549325eca4b79be5c3861c96f23b2"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:10:52.956411 8583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 08:10:52.960215 8597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.961845 8583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:52.962046 8583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0ac549325eca4b79be5c3861c96f23b2"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:10:52.962260 8583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:52.993769 8583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:52.994742 8583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:52.995039 8583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:52.996065 8583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:52.997730 8583 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:52.997833 8583 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:52.997960 8583 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:52.998034 8583 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.029915 8583 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.67:42103
I20240501 08:10:53.029994 8709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.67:42103 every 8 connection(s)
I20240501 08:10:53.031489 8583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:10:53.032531 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 8583
I20240501 08:10:53.032727 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:10:53.036276 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.68:0
--local_ip_for_outbound_sockets=127.6.56.68
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:53.044937 8710 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:10:53.045274 8710 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:53.046052 8710 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:10:53.047616 8263 ts_manager.cc:194] Registered new tserver with Master: 0ac549325eca4b79be5c3861c96f23b2 (127.6.56.67:42103)
I20240501 08:10:53.048543 8263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.67:45845
I20240501 08:10:53.127036 8449 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
W20240501 08:10:53.372419 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:53.372733 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:53.372881 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:53.380399 8713 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:53.380544 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:53.380710 8713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.68
I20240501 08:10:53.390053 8713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.68
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:53.391417 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:53.393191 8713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:53.409165 8721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.409811 8718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.410698 8719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:53.411523 8713 server_base.cc:1034] running on GCE node
I20240501 08:10:53.412343 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:53.413748 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:53.414959 8713 hybrid_clock.cc:648] HybridClock initialized: now 1714551053414885 us; error 72 us; skew 500 ppm
I20240501 08:10:53.415300 8713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:53.418639 8713 webserver.cc:460] Webserver started at http://127.6.56.68:42137/ using document root <none> and password file <none>
I20240501 08:10:53.419730 8713 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:53.419875 8713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:53.420279 8713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:53.423986 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "74bfd657aa5544d1a0925893d9121c7c"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-f1nk"
I20240501 08:10:53.425033 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "74bfd657aa5544d1a0925893d9121c7c"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-f1nk"
I20240501 08:10:53.432693 8713 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240501 08:10:53.437992 8727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.439590 8713 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:10:53.439801 8713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "74bfd657aa5544d1a0925893d9121c7c"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-f1nk"
I20240501 08:10:53.440058 8713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:53.460911 8713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:53.461871 8713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:53.462159 8713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:53.463088 8713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:53.464617 8713 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:53.464717 8713 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.464833 8713 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:53.464927 8713 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.495661 8713 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.68:38709
I20240501 08:10:53.495721 8839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.68:38709 every 8 connection(s)
I20240501 08:10:53.497040 8713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:10:53.505601 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 8713
I20240501 08:10:53.505795 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:10:53.509389 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.69:0
--local_ip_for_outbound_sockets=127.6.56.69
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:53.510634 8840 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:10:53.510954 8840 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:53.511725 8840 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:10:53.513108 8263 ts_manager.cc:194] Registered new tserver with Master: 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68:38709)
I20240501 08:10:53.513888 8263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.68:37533
I20240501 08:10:53.576576 8580 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
W20240501 08:10:53.847249 8843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:53.847551 8843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:53.847671 8843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:53.855296 8843 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:53.855432 8843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:53.855587 8843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.69
I20240501 08:10:53.864831 8843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.69
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:53.866302 8843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:53.868216 8843 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:53.881093 8849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.881711 8851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:53.882700 8848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:53.883525 8843 server_base.cc:1034] running on GCE node
I20240501 08:10:53.884179 8843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:53.885223 8843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:53.886430 8843 hybrid_clock.cc:648] HybridClock initialized: now 1714551053886383 us; error 40 us; skew 500 ppm
I20240501 08:10:53.886770 8843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:53.889289 8843 webserver.cc:460] Webserver started at http://127.6.56.69:36651/ using document root <none> and password file <none>
I20240501 08:10:53.890067 8843 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:53.890187 8843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:53.890507 8843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:53.893326 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e88366e322ed40e7bcff964b07cdeaf2"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-f1nk"
I20240501 08:10:53.894145 8843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e88366e322ed40e7bcff964b07cdeaf2"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-f1nk"
I20240501 08:10:53.899569 8843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 08:10:53.903214 8857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.904812 8843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:10:53.904989 8843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e88366e322ed40e7bcff964b07cdeaf2"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-f1nk"
I20240501 08:10:53.905205 8843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:53.927009 8843 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:53.927991 8843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:53.928261 8843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:53.929337 8843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:53.930891 8843 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:53.930994 8843 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.931228 8843 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:53.931358 8843 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:53.963610 8843 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.69:33185
I20240501 08:10:53.963656 8969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.69:33185 every 8 connection(s)
I20240501 08:10:53.965061 8843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:10:53.968379 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 8843
I20240501 08:10:53.968561 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:10:53.971526 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.70:0
--local_ip_for_outbound_sockets=127.6.56.70
--webserver_interface=127.6.56.70
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:53.979162 8970 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:10:53.979506 8970 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:53.980337 8970 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:10:53.981871 8263 ts_manager.cc:194] Registered new tserver with Master: e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69:33185)
I20240501 08:10:53.982677 8263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.69:37139
I20240501 08:10:54.051329 8710 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
W20240501 08:10:54.312726 8973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:54.313023 8973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:54.313109 8973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:54.320607 8973 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:54.320737 8973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:54.320852 8973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.70
I20240501 08:10:54.330053 8973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.56.70
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.70
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:54.331389 8973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:54.333163 8973 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:54.345963 8979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:54.346902 8978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:54.347553 8981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:54.348686 8973 server_base.cc:1034] running on GCE node
I20240501 08:10:54.349305 8973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:54.350224 8973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:54.351416 8973 hybrid_clock.cc:648] HybridClock initialized: now 1714551054351361 us; error 48 us; skew 500 ppm
I20240501 08:10:54.351706 8973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:54.354184 8973 webserver.cc:460] Webserver started at http://127.6.56.70:43901/ using document root <none> and password file <none>
I20240501 08:10:54.354904 8973 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:54.355000 8973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:54.355276 8973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:54.357992 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "ac552fa18d104068846a832248e27bcc"
format_stamp: "Formatted at 2024-05-01 08:10:54 on dist-test-slave-f1nk"
I20240501 08:10:54.358740 8973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "ac552fa18d104068846a832248e27bcc"
format_stamp: "Formatted at 2024-05-01 08:10:54 on dist-test-slave-f1nk"
I20240501 08:10:54.364341 8973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 08:10:54.368090 8987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.369669 8973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240501 08:10:54.369841 8973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ac552fa18d104068846a832248e27bcc"
format_stamp: "Formatted at 2024-05-01 08:10:54 on dist-test-slave-f1nk"
I20240501 08:10:54.370051 8973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:54.399195 8973 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:54.400156 8973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:54.400481 8973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:54.401489 8973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:54.403059 8973 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:54.403165 8973 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.403292 8973 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:54.403379 8973 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.435928 8973 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.70:33819
I20240501 08:10:54.435983 9099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.70:33819 every 8 connection(s)
I20240501 08:10:54.437314 8973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:10:54.440670 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 8973
I20240501 08:10:54.440847 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:10:54.443835 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.71:0
--local_ip_for_outbound_sockets=127.6.56.71
--webserver_interface=127.6.56.71
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:10:54.451728 9100 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:10:54.452108 9100 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:54.453163 9100 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:10:54.454859 8263 ts_manager.cc:194] Registered new tserver with Master: ac552fa18d104068846a832248e27bcc (127.6.56.70:33819)
I20240501 08:10:54.455718 8263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.70:38323
I20240501 08:10:54.516438 8840 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
W20240501 08:10:54.795737 9103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:54.796077 9103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:10:54.796207 9103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:54.803617 9103 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:10:54.803778 9103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:10:54.803965 9103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.71
I20240501 08:10:54.814102 9103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.56.71
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.71
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:10:54.815487 9103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:10:54.817562 9103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:10:54.831110 9108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:54.832054 9109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:10:54.832834 9111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:10:54.833715 9103 server_base.cc:1034] running on GCE node
I20240501 08:10:54.834365 9103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:10:54.835314 9103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:10:54.836506 9103 hybrid_clock.cc:648] HybridClock initialized: now 1714551054836447 us; error 54 us; skew 500 ppm
I20240501 08:10:54.836809 9103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:10:54.838995 9103 webserver.cc:460] Webserver started at http://127.6.56.71:38887/ using document root <none> and password file <none>
I20240501 08:10:54.839787 9103 fs_manager.cc:359] Metadata directory not provided
I20240501 08:10:54.839895 9103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:10:54.840250 9103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:10:54.842901 9103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6b0e60a3c4f74fdba1981d86fec4c529"
format_stamp: "Formatted at 2024-05-01 08:10:54 on dist-test-slave-f1nk"
I20240501 08:10:54.843649 9103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6b0e60a3c4f74fdba1981d86fec4c529"
format_stamp: "Formatted at 2024-05-01 08:10:54 on dist-test-slave-f1nk"
I20240501 08:10:54.849663 9103 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20240501 08:10:54.853199 9117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.854740 9103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:10:54.854910 9103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6b0e60a3c4f74fdba1981d86fec4c529"
format_stamp: "Formatted at 2024-05-01 08:10:54 on dist-test-slave-f1nk"
I20240501 08:10:54.855129 9103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:10:54.874403 9103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:10:54.875881 9103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:10:54.876159 9103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:10:54.877146 9103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:10:54.878661 9103 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:10:54.878759 9103 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.878875 9103 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:10:54.878945 9103 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:10:54.914921 9103 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.71:42449
I20240501 08:10:54.914969 9229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.71:42449 every 8 connection(s)
I20240501 08:10:54.916329 9103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:10:54.924168 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 9103
I20240501 08:10:54.924899 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:10:54.929428 9230 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:10:54.929697 9230 heartbeater.cc:464] Registering TS with master...
I20240501 08:10:54.930397 9230 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:10:54.931638 8263 ts_manager.cc:194] Registered new tserver with Master: 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71:42449)
I20240501 08:10:54.932502 8263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.71:45617
I20240501 08:10:54.941457 6369 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 08:10:54.958204 8263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52440:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 08:10:54.985534 8970 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
I20240501 08:10:54.986287 8645 tablet_service.cc:1467] Processing CreateTablet for tablet 197210fe0ae04e28be414b5a08d15b7c (DEFAULT_TABLE table=TestTable [id=20fd916de0a049bca49360247c56d002]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:54.987697 8645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 197210fe0ae04e28be414b5a08d15b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:54.998375 9244 tablet_bootstrap.cc:492] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Bootstrap starting.
I20240501 08:10:55.001312 9244 tablet_bootstrap.cc:654] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:55.002302 9244 log.cc:824] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:55.004882 9244 tablet_bootstrap.cc:492] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: No bootstrap required, opened a new log
I20240501 08:10:55.005126 9244 ts_tablet_manager.cc:1367] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:10:55.011615 9244 raft_consensus.cc:348] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:10:55.011967 9244 raft_consensus.cc:374] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:55.012061 9244 raft_consensus.cc:729] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac549325eca4b79be5c3861c96f23b2, State: Initialized, Role: FOLLOWER
I20240501 08:10:55.012712 9244 consensus_queue.cc:260] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:10:55.012914 9244 raft_consensus.cc:388] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:55.013036 9244 raft_consensus.cc:482] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:55.013212 9244 raft_consensus.cc:3037] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:55.015149 9244 raft_consensus.cc:504] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:10:55.015611 9244 leader_election.cc:302] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac549325eca4b79be5c3861c96f23b2; no voters:
I20240501 08:10:55.016160 9244 leader_election.cc:288] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:55.016340 9246 raft_consensus.cc:2781] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:55.016933 9246 raft_consensus.cc:686] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac549325eca4b79be5c3861c96f23b2, State: Running, Role: LEADER
I20240501 08:10:55.017479 9246 consensus_queue.cc:237] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:10:55.017863 9244 ts_tablet_manager.cc:1398] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20240501 08:10:55.022455 8263 catalog_manager.cc:5484] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ac549325eca4b79be5c3861c96f23b2 (127.6.56.67). New cstate: current_term: 1 leader_uuid: "0ac549325eca4b79be5c3861c96f23b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } } }
I20240501 08:10:55.032207 6369 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 08:10:55.034574 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fdc83c70fcc04efda1a05f9d8199b9e9 to finish bootstrapping
I20240501 08:10:55.045302 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2631692a8bb44abfb7d05022e6ae9b94 to finish bootstrapping
I20240501 08:10:55.053535 6369 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0ac549325eca4b79be5c3861c96f23b2 to finish bootstrapping
I20240501 08:10:55.061496 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 74bfd657aa5544d1a0925893d9121c7c to finish bootstrapping
I20240501 08:10:55.070248 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e88366e322ed40e7bcff964b07cdeaf2 to finish bootstrapping
I20240501 08:10:55.079674 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ac552fa18d104068846a832248e27bcc to finish bootstrapping
I20240501 08:10:55.087731 6369 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6b0e60a3c4f74fdba1981d86fec4c529 to finish bootstrapping
I20240501 08:10:55.097054 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 8583
I20240501 08:10:55.098045 8705 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:55.184056 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 8583
I20240501 08:10:55.200542 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 8713
I20240501 08:10:55.201663 8835 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:55.281553 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 8713
I20240501 08:10:55.297755 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 8843
I20240501 08:10:55.298867 8965 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:55.380578 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 8843
I20240501 08:10:55.396416 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 8973
I20240501 08:10:55.397527 9095 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:55.479547 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 8973
I20240501 08:10:55.495499 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 9103
I20240501 08:10:55.496701 9225 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:10:55.578264 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 9103
I20240501 08:10:59.346673 8262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52440:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:10:59.378930 8515 tablet_service.cc:1467] Processing CreateTablet for tablet 33975146f8f5488ab1818546e7cadca0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d90eb86e887478c9f151ae302c690e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.379386 8384 tablet_service.cc:1467] Processing CreateTablet for tablet 532f21629fae48408e569bdcd04c2cbd (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d90eb86e887478c9f151ae302c690e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.379341 8383 tablet_service.cc:1467] Processing CreateTablet for tablet e9cfe3dfff4848aabb0c7b420717130c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7d90eb86e887478c9f151ae302c690e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:10:59.380649 8515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33975146f8f5488ab1818546e7cadca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.380956 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9cfe3dfff4848aabb0c7b420717130c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.381371 8384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 532f21629fae48408e569bdcd04c2cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:10:59.391551 9271 tablet_bootstrap.cc:492] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94: Bootstrap starting.
I20240501 08:10:59.394124 9272 tablet_bootstrap.cc:492] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: Bootstrap starting.
I20240501 08:10:59.394737 9271 tablet_bootstrap.cc:654] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.396091 9271 log.cc:824] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:59.397579 9272 tablet_bootstrap.cc:654] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.398811 9272 log.cc:824] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: Log is configured to *not* fsync() on all Append() calls
I20240501 08:10:59.399150 9271 tablet_bootstrap.cc:492] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94: No bootstrap required, opened a new log
I20240501 08:10:59.399467 9271 ts_tablet_manager.cc:1367] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 08:10:59.401827 9272 tablet_bootstrap.cc:492] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: No bootstrap required, opened a new log
I20240501 08:10:59.402107 9272 ts_tablet_manager.cc:1367] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 08:10:59.408452 9271 raft_consensus.cc:348] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:10:59.408893 9271 raft_consensus.cc:374] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.408998 9271 raft_consensus.cc:729] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.409752 9271 consensus_queue.cc:260] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:10:59.410004 9271 raft_consensus.cc:388] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:59.410152 9271 raft_consensus.cc:482] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:59.410310 9271 raft_consensus.cc:3037] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.410897 9272 raft_consensus.cc:348] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:10:59.411291 9272 raft_consensus.cc:374] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.411391 9272 raft_consensus.cc:729] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.412056 9272 consensus_queue.cc:260] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:10:59.412338 9272 raft_consensus.cc:388] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:59.412472 9272 raft_consensus.cc:482] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:59.412626 9272 raft_consensus.cc:3037] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.413264 9271 raft_consensus.cc:504] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:10:59.413849 9271 leader_election.cc:302] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94; no voters:
I20240501 08:10:59.414516 9271 leader_election.cc:288] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:59.414638 9275 raft_consensus.cc:2781] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:59.415504 9272 raft_consensus.cc:504] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:10:59.416038 9272 leader_election.cc:302] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:10:59.416131 9275 raft_consensus.cc:686] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Leader. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:10:59.416358 9271 ts_tablet_manager.cc:1398] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent starting tablet: real 0.017s user 0.019s sys 0.000s
I20240501 08:10:59.416751 9272 leader_election.cc:288] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:59.416714 9275 consensus_queue.cc:237] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:10:59.416906 9278 raft_consensus.cc:2781] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:59.418392 9272 ts_tablet_manager.cc:1398] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
I20240501 08:10:59.418723 9272 tablet_bootstrap.cc:492] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: Bootstrap starting.
I20240501 08:10:59.419970 9272 tablet_bootstrap.cc:654] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: Neither blocks nor log segments found. Creating new log.
I20240501 08:10:59.422004 9272 tablet_bootstrap.cc:492] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: No bootstrap required, opened a new log
I20240501 08:10:59.422165 9272 ts_tablet_manager.cc:1367] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:10:59.423158 9272 raft_consensus.cc:348] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:10:59.423347 9272 raft_consensus.cc:374] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:10:59.423451 9272 raft_consensus.cc:729] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Initialized, Role: FOLLOWER
I20240501 08:10:59.423789 9272 consensus_queue.cc:260] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:10:59.423970 9272 raft_consensus.cc:388] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:10:59.423951 9278 raft_consensus.cc:686] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Leader. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:10:59.424135 9272 raft_consensus.cc:482] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:10:59.424257 9272 raft_consensus.cc:3037] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:10:59.424664 9278 consensus_queue.cc:237] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:10:59.427202 9272 raft_consensus.cc:504] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:10:59.427515 9272 leader_election.cc:302] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:10:59.427713 9272 leader_election.cc:288] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:10:59.427968 9279 raft_consensus.cc:2781] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:10:59.428537 9279 raft_consensus.cc:686] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Leader. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:10:59.428864 9279 consensus_queue.cc:237] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:10:59.429844 8262 catalog_manager.cc:5484] T 33975146f8f5488ab1818546e7cadca0 P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } }
I20240501 08:10:59.430056 9272 ts_tablet_manager.cc:1398] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20240501 08:10:59.431282 8263 catalog_manager.cc:5484] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } }
I20240501 08:10:59.440414 8262 catalog_manager.cc:5484] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } }
W20240501 08:10:59.792812 9289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:10:59.793162 9289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:10:59.800894 9289 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 154.371 ms
time per row: 0.0771853 ms
I20240501 08:11:00.110275 8262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52440:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:00.119459 8515 tablet_service.cc:1467] Processing CreateTablet for tablet 833a0a7dc6a047b19808085569e89368 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a622f24ae9504f47acdb022fb9714306]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:00.119553 8384 tablet_service.cc:1467] Processing CreateTablet for tablet e836061c829848caa03078113b94c44e (DEFAULT_TABLE table=rebalance_test_table_1 [id=a622f24ae9504f47acdb022fb9714306]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:00.119796 8514 tablet_service.cc:1467] Processing CreateTablet for tablet 3cc149e11b784b929ce06059690b4a7e (DEFAULT_TABLE table=rebalance_test_table_1 [id=a622f24ae9504f47acdb022fb9714306]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:00.119976 8515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833a0a7dc6a047b19808085569e89368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:00.119973 8384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e836061c829848caa03078113b94c44e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:00.120285 8514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cc149e11b784b929ce06059690b4a7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:00.127197 9336 tablet_bootstrap.cc:492] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9: Bootstrap starting.
I20240501 08:11:00.128137 9337 tablet_bootstrap.cc:492] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: Bootstrap starting.
I20240501 08:11:00.129016 9336 tablet_bootstrap.cc:654] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:00.129695 9337 tablet_bootstrap.cc:654] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:00.131163 9336 tablet_bootstrap.cc:492] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9: No bootstrap required, opened a new log
I20240501 08:11:00.131441 9336 ts_tablet_manager.cc:1367] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:00.132658 9337 tablet_bootstrap.cc:492] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: No bootstrap required, opened a new log
I20240501 08:11:00.132640 9336 raft_consensus.cc:348] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:00.132779 9336 raft_consensus.cc:374] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:00.132808 9337 ts_tablet_manager.cc:1367] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:11:00.132862 9336 raft_consensus.cc:729] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Initialized, Role: FOLLOWER
I20240501 08:11:00.133118 9336 consensus_queue.cc:260] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:00.133280 9336 raft_consensus.cc:388] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:00.133358 9336 raft_consensus.cc:482] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:00.133481 9336 raft_consensus.cc:3037] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:00.133669 9337 raft_consensus.cc:348] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:00.133838 9337 raft_consensus.cc:374] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:00.133935 9337 raft_consensus.cc:729] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Initialized, Role: FOLLOWER
I20240501 08:11:00.134186 9337 consensus_queue.cc:260] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:00.134380 9337 raft_consensus.cc:388] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:00.134469 9337 raft_consensus.cc:482] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:00.134581 9337 raft_consensus.cc:3037] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:00.135766 9336 raft_consensus.cc:504] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:00.135967 9336 leader_election.cc:302] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:00.136119 9336 leader_election.cc:288] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:00.136235 9322 raft_consensus.cc:2781] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:00.136605 9336 ts_tablet_manager.cc:1398] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 08:11:00.136888 9337 raft_consensus.cc:504] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:00.136960 9322 raft_consensus.cc:686] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Leader. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:00.137109 9337 leader_election.cc:302] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94; no voters:
I20240501 08:11:00.137288 9337 leader_election.cc:288] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:00.137373 9329 raft_consensus.cc:2781] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:00.137261 9322 consensus_queue.cc:237] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:00.138128 9337 ts_tablet_manager.cc:1398] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:00.138479 9337 tablet_bootstrap.cc:492] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94: Bootstrap starting.
I20240501 08:11:00.140702 9337 tablet_bootstrap.cc:654] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:00.143494 9337 tablet_bootstrap.cc:492] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94: No bootstrap required, opened a new log
I20240501 08:11:00.143702 9337 ts_tablet_manager.cc:1367] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:11:00.144757 9337 raft_consensus.cc:348] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:00.145018 9337 raft_consensus.cc:374] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:00.145115 9337 raft_consensus.cc:729] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Initialized, Role: FOLLOWER
I20240501 08:11:00.145431 9337 consensus_queue.cc:260] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:00.145638 9337 raft_consensus.cc:388] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:00.145795 9337 raft_consensus.cc:482] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:00.145928 9337 raft_consensus.cc:3037] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:00.149832 9337 raft_consensus.cc:504] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:00.150177 9337 leader_election.cc:302] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94; no voters:
I20240501 08:11:00.150511 9337 leader_election.cc:288] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:00.150666 9341 raft_consensus.cc:2781] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:00.151114 9337 ts_tablet_manager.cc:1398] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 08:11:00.152796 9329 raft_consensus.cc:686] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Leader. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:00.153215 9329 consensus_queue.cc:237] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:00.154385 9341 raft_consensus.cc:686] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Leader. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:00.154695 9341 consensus_queue.cc:237] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:00.155330 8258 catalog_manager.cc:5484] T e836061c829848caa03078113b94c44e P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } }
I20240501 08:11:00.158280 8262 catalog_manager.cc:5484] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } }
I20240501 08:11:00.162667 8258 catalog_manager.cc:5484] T 3cc149e11b784b929ce06059690b4a7e P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } }
W20240501 08:11:00.510669 9348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:00.511001 9348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:00.518509 9348 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 143.934 ms
time per row: 0.071967 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:01.725703 8261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52440:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:01.734593 8384 tablet_service.cc:1467] Processing CreateTablet for tablet d227b998061e46f59653167740f87fd0 (DEFAULT_TABLE table=rebalance_test_table_2 [id=dee2da36183e48bf99429c14839719ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.734694 8515 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4f5ffc5f44469d9e65db69546d836f (DEFAULT_TABLE table=rebalance_test_table_2 [id=dee2da36183e48bf99429c14839719ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.734983 8383 tablet_service.cc:1467] Processing CreateTablet for tablet d71f0f9eeffd476ebd70487ff9a93d7c (DEFAULT_TABLE table=rebalance_test_table_2 [id=dee2da36183e48bf99429c14839719ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:01.735105 8515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4f5ffc5f44469d9e65db69546d836f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.735121 8384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d227b998061e46f59653167740f87fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.735363 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71f0f9eeffd476ebd70487ff9a93d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:01.741708 9398 tablet_bootstrap.cc:492] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: Bootstrap starting.
I20240501 08:11:01.742063 9397 tablet_bootstrap.cc:492] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: Bootstrap starting.
I20240501 08:11:01.743507 9398 tablet_bootstrap.cc:654] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.743614 9397 tablet_bootstrap.cc:654] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.745368 9398 tablet_bootstrap.cc:492] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: No bootstrap required, opened a new log
I20240501 08:11:01.745525 9398 ts_tablet_manager.cc:1367] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:01.745568 9397 tablet_bootstrap.cc:492] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: No bootstrap required, opened a new log
I20240501 08:11:01.745702 9397 ts_tablet_manager.cc:1367] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:01.746117 9398 raft_consensus.cc:348] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:01.746275 9398 raft_consensus.cc:374] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.746351 9398 raft_consensus.cc:729] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.746449 9397 raft_consensus.cc:348] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:01.746538 9398 consensus_queue.cc:260] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:01.746645 9397 raft_consensus.cc:374] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.746728 9398 raft_consensus.cc:388] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:01.746734 9397 raft_consensus.cc:729] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.746806 9398 raft_consensus.cc:482] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:01.746913 9398 raft_consensus.cc:3037] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:01.746935 9397 consensus_queue.cc:260] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:01.747077 9397 raft_consensus.cc:388] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:01.747133 9397 raft_consensus.cc:482] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:01.747207 9397 raft_consensus.cc:3037] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:01.748869 9398 raft_consensus.cc:504] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:01.749120 9398 leader_election.cc:302] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94; no voters:
I20240501 08:11:01.749164 9397 raft_consensus.cc:504] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:01.749377 9397 leader_election.cc:302] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:01.749541 9398 leader_election.cc:288] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:01.749748 9401 raft_consensus.cc:2781] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:01.750352 9398 ts_tablet_manager.cc:1398] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240501 08:11:01.750689 9401 raft_consensus.cc:686] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Leader. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:01.750864 9397 leader_election.cc:288] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:01.750978 9402 raft_consensus.cc:2781] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:01.750975 9401 consensus_queue.cc:237] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:01.751976 9397 ts_tablet_manager.cc:1398] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20240501 08:11:01.752331 9397 tablet_bootstrap.cc:492] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: Bootstrap starting.
I20240501 08:11:01.754086 9397 tablet_bootstrap.cc:654] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:01.754462 8261 catalog_manager.cc:5484] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } }
I20240501 08:11:01.758674 9402 raft_consensus.cc:686] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Leader. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:01.759014 9402 consensus_queue.cc:237] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:01.760640 9397 tablet_bootstrap.cc:492] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: No bootstrap required, opened a new log
I20240501 08:11:01.760826 9397 ts_tablet_manager.cc:1367] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 08:11:01.761607 9397 raft_consensus.cc:348] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:01.761766 9397 raft_consensus.cc:374] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:01.761852 9397 raft_consensus.cc:729] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Initialized, Role: FOLLOWER
I20240501 08:11:01.762074 9397 consensus_queue.cc:260] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:01.762228 9397 raft_consensus.cc:388] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:01.762331 9397 raft_consensus.cc:482] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:01.762450 9397 raft_consensus.cc:3037] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:01.764740 9397 raft_consensus.cc:504] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:01.764968 9397 leader_election.cc:302] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:01.765192 9397 leader_election.cc:288] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:01.765256 9402 raft_consensus.cc:2781] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:01.765532 9402 raft_consensus.cc:686] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Leader. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:01.765717 9397 ts_tablet_manager.cc:1398] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:01.765801 9402 consensus_queue.cc:237] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:01.766217 8261 catalog_manager.cc:5484] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } }
I20240501 08:11:01.775317 8261 catalog_manager.cc:5484] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } }
W20240501 08:11:02.112680 9414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:02.113045 9414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:02.120719 9414 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 119.738 ms
time per row: 0.0598688 ms
I20240501 08:11:02.387403 8259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52440:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:02.396229 8515 tablet_service.cc:1467] Processing CreateTablet for tablet e0ebe3885c6f4e3794bb61abf33dbb97 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9fbdcb332a0a4e90952a76a63564ab9c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:02.396323 8514 tablet_service.cc:1467] Processing CreateTablet for tablet 80636827cb8c427c8317f676d8790c70 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9fbdcb332a0a4e90952a76a63564ab9c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:02.396569 8384 tablet_service.cc:1467] Processing CreateTablet for tablet 9e92c6c0f9114282b832ecaddffea60a (DEFAULT_TABLE table=rebalance_test_table_3 [id=9fbdcb332a0a4e90952a76a63564ab9c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:02.396791 8515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ebe3885c6f4e3794bb61abf33dbb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.396987 8514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80636827cb8c427c8317f676d8790c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.396984 8384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e92c6c0f9114282b832ecaddffea60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:02.402817 9455 tablet_bootstrap.cc:492] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: Bootstrap starting.
I20240501 08:11:02.403856 9456 tablet_bootstrap.cc:492] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9: Bootstrap starting.
I20240501 08:11:02.404474 9455 tablet_bootstrap.cc:654] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.405361 9456 tablet_bootstrap.cc:654] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.406741 9455 tablet_bootstrap.cc:492] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: No bootstrap required, opened a new log
I20240501 08:11:02.406888 9455 ts_tablet_manager.cc:1367] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:11:02.407166 9456 tablet_bootstrap.cc:492] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9: No bootstrap required, opened a new log
I20240501 08:11:02.407297 9456 ts_tablet_manager.cc:1367] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:02.407675 9455 raft_consensus.cc:348] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:02.407884 9455 raft_consensus.cc:374] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.407984 9455 raft_consensus.cc:729] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.408037 9456 raft_consensus.cc:348] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:02.408217 9456 raft_consensus.cc:374] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.408339 9456 raft_consensus.cc:729] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.408234 9455 consensus_queue.cc:260] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:02.408457 9455 raft_consensus.cc:388] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:02.408540 9455 raft_consensus.cc:482] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:02.408571 9456 consensus_queue.cc:260] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:02.408656 9455 raft_consensus.cc:3037] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.408759 9456 raft_consensus.cc:388] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:02.408841 9456 raft_consensus.cc:482] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:02.408974 9456 raft_consensus.cc:3037] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.411387 9455 raft_consensus.cc:504] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:02.411614 9456 raft_consensus.cc:504] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:02.411664 9455 leader_election.cc:302] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94; no voters:
I20240501 08:11:02.411852 9455 leader_election.cc:288] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:02.411845 9456 leader_election.cc:302] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:02.411962 9404 raft_consensus.cc:2781] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:02.412041 9456 leader_election.cc:288] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:02.412144 9406 raft_consensus.cc:2781] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:02.412811 9456 ts_tablet_manager.cc:1398] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20240501 08:11:02.413619 9455 ts_tablet_manager.cc:1398] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20240501 08:11:02.413616 9404 raft_consensus.cc:686] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Leader. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:02.413944 9455 tablet_bootstrap.cc:492] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: Bootstrap starting.
I20240501 08:11:02.414039 9404 consensus_queue.cc:237] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:02.414204 9406 raft_consensus.cc:686] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Leader. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:02.414489 9406 consensus_queue.cc:237] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:02.415625 9455 tablet_bootstrap.cc:654] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:02.418170 8259 catalog_manager.cc:5484] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } }
I20240501 08:11:02.418592 9455 tablet_bootstrap.cc:492] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: No bootstrap required, opened a new log
I20240501 08:11:02.418756 9455 ts_tablet_manager.cc:1367] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:11:02.419477 9455 raft_consensus.cc:348] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:02.419669 9455 raft_consensus.cc:374] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:02.419749 9455 raft_consensus.cc:729] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Initialized, Role: FOLLOWER
I20240501 08:11:02.419974 9455 consensus_queue.cc:260] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:02.420130 9455 raft_consensus.cc:388] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:02.420226 9455 raft_consensus.cc:482] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:02.420352 9455 raft_consensus.cc:3037] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:02.422031 8255 catalog_manager.cc:5484] T 9e92c6c0f9114282b832ecaddffea60a P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } }
I20240501 08:11:02.423166 9455 raft_consensus.cc:504] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:02.423370 9455 leader_election.cc:302] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94; no voters:
I20240501 08:11:02.423550 9455 leader_election.cc:288] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:02.423600 9459 raft_consensus.cc:2781] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:02.423784 9459 raft_consensus.cc:686] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Leader. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:02.424021 9455 ts_tablet_manager.cc:1398] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:11:02.424075 9459 consensus_queue.cc:237] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:02.426261 8259 catalog_manager.cc:5484] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } }
W20240501 08:11:02.765352 9466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:02.765692 9466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:02.773615 9466 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 125.125 ms
time per row: 0.0625625 ms
I20240501 08:11:03.039044 8259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52440:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:03.047551 8384 tablet_service.cc:1467] Processing CreateTablet for tablet fe8bf89936b04faca8c7ab15ce402e8f (DEFAULT_TABLE table=rebalance_test_table_4 [id=62a46f9934254f7a93b4a1ec0eef395b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.047667 8515 tablet_service.cc:1467] Processing CreateTablet for tablet 1f6207df29954aa3bfbc81304bd38827 (DEFAULT_TABLE table=rebalance_test_table_4 [id=62a46f9934254f7a93b4a1ec0eef395b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.048046 8515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6207df29954aa3bfbc81304bd38827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.048058 8384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8bf89936b04faca8c7ab15ce402e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.048327 8383 tablet_service.cc:1467] Processing CreateTablet for tablet 493df0048a8740f190d7d126e70e8bc5 (DEFAULT_TABLE table=rebalance_test_table_4 [id=62a46f9934254f7a93b4a1ec0eef395b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:03.048676 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493df0048a8740f190d7d126e70e8bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:03.054349 9507 tablet_bootstrap.cc:492] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: Bootstrap starting.
I20240501 08:11:03.054970 9508 tablet_bootstrap.cc:492] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: Bootstrap starting.
I20240501 08:11:03.055881 9507 tablet_bootstrap.cc:654] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.056368 9508 tablet_bootstrap.cc:654] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.057904 9507 tablet_bootstrap.cc:492] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: No bootstrap required, opened a new log
I20240501 08:11:03.058033 9507 ts_tablet_manager.cc:1367] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:03.058288 9508 tablet_bootstrap.cc:492] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: No bootstrap required, opened a new log
I20240501 08:11:03.058442 9508 ts_tablet_manager.cc:1367] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:03.058696 9507 raft_consensus.cc:348] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:03.058844 9507 raft_consensus.cc:374] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.058928 9507 raft_consensus.cc:729] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.059057 9508 raft_consensus.cc:348] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:03.059141 9507 consensus_queue.cc:260] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:03.059232 9508 raft_consensus.cc:374] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.059298 9507 raft_consensus.cc:388] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:03.059331 9508 raft_consensus.cc:729] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.059374 9507 raft_consensus.cc:482] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:03.059486 9507 raft_consensus.cc:3037] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:03.059584 9508 consensus_queue.cc:260] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:03.059744 9508 raft_consensus.cc:388] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:03.059819 9508 raft_consensus.cc:482] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:03.059921 9508 raft_consensus.cc:3037] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:03.061583 9507 raft_consensus.cc:504] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:03.061822 9507 leader_election.cc:302] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:03.061981 9508 raft_consensus.cc:504] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:03.062062 9507 leader_election.cc:288] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:03.062152 9406 raft_consensus.cc:2781] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:03.062270 9508 leader_election.cc:302] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94; no voters:
I20240501 08:11:03.062609 9460 raft_consensus.cc:2781] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:03.062767 9460 raft_consensus.cc:686] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Leader. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:03.062856 9507 ts_tablet_manager.cc:1398] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:03.063107 9507 tablet_bootstrap.cc:492] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: Bootstrap starting.
I20240501 08:11:03.063122 9460 consensus_queue.cc:237] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:03.064674 9507 tablet_bootstrap.cc:654] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:03.065107 9508 leader_election.cc:288] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:03.065917 9508 ts_tablet_manager.cc:1398] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:11:03.068928 8259 catalog_manager.cc:5484] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } }
I20240501 08:11:03.074326 9507 tablet_bootstrap.cc:492] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: No bootstrap required, opened a new log
I20240501 08:11:03.074535 9406 raft_consensus.cc:686] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Leader. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:03.074601 9507 ts_tablet_manager.cc:1367] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 08:11:03.074841 9406 consensus_queue.cc:237] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:03.075572 9507 raft_consensus.cc:348] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:03.075786 9507 raft_consensus.cc:374] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:03.075876 9507 raft_consensus.cc:729] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Initialized, Role: FOLLOWER
I20240501 08:11:03.076121 9507 consensus_queue.cc:260] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:03.076324 9507 raft_consensus.cc:388] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:03.076440 9507 raft_consensus.cc:482] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:03.076622 9507 raft_consensus.cc:3037] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:03.077520 8259 catalog_manager.cc:5484] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } }
I20240501 08:11:03.079324 9507 raft_consensus.cc:504] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:03.079560 9507 leader_election.cc:302] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:03.080513 9507 leader_election.cc:288] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:03.080957 9511 raft_consensus.cc:2781] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:03.081081 9511 raft_consensus.cc:686] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Leader. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:03.081336 9511 consensus_queue.cc:237] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:03.083102 9507 ts_tablet_manager.cc:1398] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: Time spent starting tablet: real 0.008s user 0.000s sys 0.005s
I20240501 08:11:03.085016 8256 catalog_manager.cc:5484] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } }
W20240501 08:11:03.428369 9518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:03.428712 9518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:03.436276 9518 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 153.992 ms
time per row: 0.0769959 ms
I20240501 08:11:03.745235 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.67:42103
--local_ip_for_outbound_sockets=127.6.56.67
--tserver_master_addrs=127.6.56.126:36943
--webserver_port=33321
--webserver_interface=127.6.56.67
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:11:04.070626 9560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:04.070951 9560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:04.071091 9560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:04.078979 9560 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:04.079142 9560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:04.079303 9560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.67
I20240501 08:11:04.089329 9560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.67:42103
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.56.67
--webserver_port=33321
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.67
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:04.090804 9560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:04.092708 9560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:04.105155 9565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:04.106048 9566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:04.106747 9568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:04.107734 9560 server_base.cc:1034] running on GCE node
I20240501 08:11:04.108454 9560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:04.109586 9560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:04.110792 9560 hybrid_clock.cc:648] HybridClock initialized: now 1714551064110737 us; error 51 us; skew 500 ppm
I20240501 08:11:04.111158 9560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:04.113894 9560 webserver.cc:460] Webserver started at http://127.6.56.67:33321/ using document root <none> and password file <none>
I20240501 08:11:04.114804 9560 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:04.114961 9560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:04.121440 9560 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 08:11:04.124375 9574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:04.125752 9560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 08:11:04.125914 9560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0ac549325eca4b79be5c3861c96f23b2"
format_stamp: "Formatted at 2024-05-01 08:10:52 on dist-test-slave-f1nk"
I20240501 08:11:04.126801 9560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:04.142143 9560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:04.143030 9560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:04.143304 9560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:04.144212 9560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:04.146170 9581 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 08:11:04.152015 9560 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 08:11:04.152419 9560 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20240501 08:11:04.152588 9560 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 08:11:04.155752 9560 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 08:11:04.155845 9560 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240501 08:11:04.156216 9581 tablet_bootstrap.cc:492] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Bootstrap starting.
I20240501 08:11:04.175693 9581 log.cc:824] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:04.181790 9581 tablet_bootstrap.cc:492] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:11:04.182760 9581 tablet_bootstrap.cc:492] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Bootstrap complete.
I20240501 08:11:04.184701 9581 ts_tablet_manager.cc:1367] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.015s
I20240501 08:11:04.192113 9560 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.67:42103
I20240501 08:11:04.192183 9688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.67:42103 every 8 connection(s)
I20240501 08:11:04.194049 9560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:11:04.197415 9581 raft_consensus.cc:348] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:11:04.198240 9581 raft_consensus.cc:729] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac549325eca4b79be5c3861c96f23b2, State: Initialized, Role: FOLLOWER
I20240501 08:11:04.198904 9581 consensus_queue.cc:260] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:11:04.199100 9581 raft_consensus.cc:388] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:04.199223 9581 raft_consensus.cc:482] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:04.199376 9581 raft_consensus.cc:3037] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:04.203950 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 9560
I20240501 08:11:04.204398 9581 raft_consensus.cc:504] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:11:04.204742 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.68:38709
--local_ip_for_outbound_sockets=127.6.56.68
--tserver_master_addrs=127.6.56.126:36943
--webserver_port=42137
--webserver_interface=127.6.56.68
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:04.205114 9581 leader_election.cc:302] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac549325eca4b79be5c3861c96f23b2; no voters:
I20240501 08:11:04.205807 9581 leader_election.cc:288] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240501 08:11:04.205978 9693 raft_consensus.cc:2781] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:04.208595 9693 raft_consensus.cc:686] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [term 2 LEADER]: Becoming Leader. State: Replica: 0ac549325eca4b79be5c3861c96f23b2, State: Running, Role: LEADER
I20240501 08:11:04.209489 9693 consensus_queue.cc:237] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:11:04.210376 9581 ts_tablet_manager.cc:1398] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2: Time spent starting tablet: real 0.026s user 0.015s sys 0.009s
I20240501 08:11:04.210667 9689 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:11:04.210932 9689 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:04.211968 9689 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:11:04.214696 8259 ts_manager.cc:194] Re-registered known tserver with Master: 0ac549325eca4b79be5c3861c96f23b2 (127.6.56.67:42103)
I20240501 08:11:04.215081 8259 catalog_manager.cc:5484] T 197210fe0ae04e28be414b5a08d15b7c P 0ac549325eca4b79be5c3861c96f23b2 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "0ac549325eca4b79be5c3861c96f23b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } } }
I20240501 08:11:04.219555 8259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.67:58429
I20240501 08:11:04.222102 9689 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
W20240501 08:11:04.531416 9696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:04.531733 9696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:04.531863 9696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:04.539672 9696 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:04.539804 9696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:04.539975 9696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.68
I20240501 08:11:04.549309 9696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.68:38709
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.56.68
--webserver_port=42137
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.68
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:04.550679 9696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:04.552501 9696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:04.567922 9707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:04.569031 9708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:04.569597 9710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:04.570338 9696 server_base.cc:1034] running on GCE node
I20240501 08:11:04.570933 9696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:04.571846 9696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:04.573045 9696 hybrid_clock.cc:648] HybridClock initialized: now 1714551064572989 us; error 43 us; skew 500 ppm
I20240501 08:11:04.573344 9696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:04.575558 9696 webserver.cc:460] Webserver started at http://127.6.56.68:42137/ using document root <none> and password file <none>
I20240501 08:11:04.576275 9696 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:04.576432 9696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:04.581527 9696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 08:11:04.584144 9716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:04.585430 9696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240501 08:11:04.585594 9696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "74bfd657aa5544d1a0925893d9121c7c"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-f1nk"
I20240501 08:11:04.586426 9696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:04.612516 9696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:04.613346 9696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:04.613618 9696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:04.614544 9696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:04.616024 9696 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:04.616109 9696 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:04.616236 9696 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:04.616323 9696 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:04.646086 9696 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.68:38709
I20240501 08:11:04.646112 9828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.68:38709 every 8 connection(s)
I20240501 08:11:04.647578 9696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:11:04.656118 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 9696
I20240501 08:11:04.656724 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.69:33185
--local_ip_for_outbound_sockets=127.6.56.69
--tserver_master_addrs=127.6.56.126:36943
--webserver_port=36651
--webserver_interface=127.6.56.69
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:04.660321 9829 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:11:04.660683 9829 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:04.661439 9829 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:11:04.662827 8259 ts_manager.cc:194] Re-registered known tserver with Master: 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68:38709)
I20240501 08:11:04.663630 8259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.68:60907
W20240501 08:11:04.986656 9832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:04.986997 9832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:04.987144 9832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:04.994792 9832 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:04.994964 9832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:04.995142 9832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.69
I20240501 08:11:05.004392 9832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.69:33185
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.56.69
--webserver_port=36651
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.69
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:05.005797 9832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:05.007582 9832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:05.019901 9838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:05.020573 9837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:05.021625 9840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:05.022467 9832 server_base.cc:1034] running on GCE node
I20240501 08:11:05.023121 9832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:05.024070 9832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:05.025280 9832 hybrid_clock.cc:648] HybridClock initialized: now 1714551065025234 us; error 46 us; skew 500 ppm
I20240501 08:11:05.025604 9832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:05.032476 9832 webserver.cc:460] Webserver started at http://127.6.56.69:36651/ using document root <none> and password file <none>
I20240501 08:11:05.033294 9832 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:05.033443 9832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:05.039281 9832 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20240501 08:11:05.042230 9846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:05.043612 9832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:05.043790 9832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e88366e322ed40e7bcff964b07cdeaf2"
format_stamp: "Formatted at 2024-05-01 08:10:53 on dist-test-slave-f1nk"
I20240501 08:11:05.044680 9832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:05.068101 9832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:05.069080 9832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:05.069391 9832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:05.070415 9832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:05.071964 9832 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:05.072044 9832 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:05.072176 9832 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:05.072232 9832 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:05.104818 9832 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.69:33185
I20240501 08:11:05.104836 9958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.69:33185 every 8 connection(s)
I20240501 08:11:05.106276 9832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:11:05.106683 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 9832
I20240501 08:11:05.107362 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.70:33819
--local_ip_for_outbound_sockets=127.6.56.70
--tserver_master_addrs=127.6.56.126:36943
--webserver_port=43901
--webserver_interface=127.6.56.70
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:05.119758 9959 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:11:05.120069 9959 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:05.120803 9959 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:11:05.122260 8259 ts_manager.cc:194] Re-registered known tserver with Master: e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69:33185)
I20240501 08:11:05.123466 8259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.69:34491
W20240501 08:11:05.438110 9962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:05.438552 9962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:05.438681 9962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:05.446347 9962 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:05.446472 9962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:05.446590 9962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.70
I20240501 08:11:05.458537 9962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.70:33819
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.56.70
--webserver_port=43901
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.70
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:05.460188 9962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:05.462080 9962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:05.474570 9968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:05.475131 9970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:05.475972 9967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:05.476882 9962 server_base.cc:1034] running on GCE node
I20240501 08:11:05.477512 9962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:05.478492 9962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:05.479679 9962 hybrid_clock.cc:648] HybridClock initialized: now 1714551065479631 us; error 41 us; skew 500 ppm
I20240501 08:11:05.479982 9962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:05.482206 9962 webserver.cc:460] Webserver started at http://127.6.56.70:43901/ using document root <none> and password file <none>
I20240501 08:11:05.482933 9962 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:05.483059 9962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:05.488462 9962 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:05.491216 9976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:05.492563 9962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:05.492715 9962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "ac552fa18d104068846a832248e27bcc"
format_stamp: "Formatted at 2024-05-01 08:10:54 on dist-test-slave-f1nk"
I20240501 08:11:05.493502 9962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:05.514842 9962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:05.515847 9962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:05.516166 9962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:05.517181 9962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:05.518772 9962 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:05.518851 9962 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:05.518976 9962 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:05.519029 9962 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:05.550908 9962 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.70:33819
I20240501 08:11:05.550951 10088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.70:33819 every 8 connection(s)
I20240501 08:11:05.552369 9962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:11:05.556813 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 9962
I20240501 08:11:05.557574 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.71:42449
--local_ip_for_outbound_sockets=127.6.56.71
--tserver_master_addrs=127.6.56.126:36943
--webserver_port=38887
--webserver_interface=127.6.56.71
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:05.566700 10089 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:11:05.567034 10089 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:05.567816 10089 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:11:05.569275 8259 ts_manager.cc:194] Re-registered known tserver with Master: ac552fa18d104068846a832248e27bcc (127.6.56.70:33819)
I20240501 08:11:05.570222 8259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.70:33939
I20240501 08:11:05.666828 9829 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
W20240501 08:11:05.903970 10092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:05.904263 10092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:05.904456 10092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:05.912338 10092 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:05.912469 10092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:05.912590 10092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.71
I20240501 08:11:05.921895 10092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.71:42449
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.56.71
--webserver_port=38887
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.71
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:05.923167 10092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:05.924844 10092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:05.937623 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:05.938091 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:05.939013 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:05.939812 10092 server_base.cc:1034] running on GCE node
I20240501 08:11:05.940430 10092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:05.941390 10092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:05.942586 10092 hybrid_clock.cc:648] HybridClock initialized: now 1714551065942541 us; error 35 us; skew 500 ppm
I20240501 08:11:05.942907 10092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:05.945173 10092 webserver.cc:460] Webserver started at http://127.6.56.71:38887/ using document root <none> and password file <none>
I20240501 08:11:05.945935 10092 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:05.946051 10092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:05.951764 10092 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240501 08:11:05.954704 10106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:05.956154 10092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:05.956341 10092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6b0e60a3c4f74fdba1981d86fec4c529"
format_stamp: "Formatted at 2024-05-01 08:10:54 on dist-test-slave-f1nk"
I20240501 08:11:05.957195 10092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:05.981796 10092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:05.982683 10092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:05.982987 10092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:05.983994 10092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:05.986114 10092 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:05.986222 10092 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:05.986333 10092 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:05.986382 10092 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:06.019802 10092 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.71:42449
I20240501 08:11:06.019842 10218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.71:42449 every 8 connection(s)
I20240501 08:11:06.021283 10092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:11:06.028249 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 10092
I20240501 08:11:06.034901 10219 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:11:06.035292 10219 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:06.036010 10219 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:11:06.037472 8259 ts_manager.cc:194] Re-registered known tserver with Master: 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71:42449)
I20240501 08:11:06.038682 8259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.71:37523
I20240501 08:11:06.126022 9959 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
I20240501 08:11:06.529335 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:06.563985 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:06.566874 9764 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:06.571269 10154 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:06.572763 10089 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
I20240501 08:11:06.621042 10024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:06.634616 8383 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:06.637070 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:06.833180 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:06.836834 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:06.848151 10154 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:06.848158 9764 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:06.850978 10024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:06.875664 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:06.897248 8383 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:06.983597 8404 consensus_queue.cc:237] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
W20240501 08:11:06.992929 8338 consensus_peers.cc:485] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 -> Peer 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68:38709): Couldn't send request to peer 74bfd657aa5544d1a0925893d9121c7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d71f0f9eeffd476ebd70487ff9a93d7c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:07.041358 10219 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
I20240501 08:11:07.546350 10320 ts_tablet_manager.cc:902] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: Initiating tablet copy from peer fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:07.547982 10320 tablet_copy_client.cc:308] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46073
I20240501 08:11:07.555038 8424 tablet_copy_service.cc:140] P fdc83c70fcc04efda1a05f9d8199b9e9: Received BeginTabletCopySession request for tablet d71f0f9eeffd476ebd70487ff9a93d7c from peer 74bfd657aa5544d1a0925893d9121c7c ({username='slave'} at 127.6.56.68:55701)
I20240501 08:11:07.555315 8424 tablet_copy_service.cc:161] P fdc83c70fcc04efda1a05f9d8199b9e9: Beginning new tablet copy session on tablet d71f0f9eeffd476ebd70487ff9a93d7c from peer 74bfd657aa5544d1a0925893d9121c7c at {username='slave'} at 127.6.56.68:55701: session id = 74bfd657aa5544d1a0925893d9121c7c-d71f0f9eeffd476ebd70487ff9a93d7c
I20240501 08:11:07.559130 8424 tablet_copy_source_session.cc:205] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:07.565656 10320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71f0f9eeffd476ebd70487ff9a93d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:07.579326 10320 tablet_copy_client.cc:783] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:07.579681 10320 tablet_copy_client.cc:647] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:07.582167 10320 tablet_copy_client.cc:523] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:07.585834 10320 tablet_bootstrap.cc:492] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: Bootstrap starting.
I20240501 08:11:07.595393 10320 log.cc:824] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:07.626930 10320 tablet_bootstrap.cc:492] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:07.627544 10320 tablet_bootstrap.cc:492] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: Bootstrap complete.
I20240501 08:11:07.628001 10320 ts_tablet_manager.cc:1367] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: Time spent bootstrapping tablet: real 0.042s user 0.039s sys 0.001s
I20240501 08:11:07.633018 10320 raft_consensus.cc:348] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:07.633929 10320 raft_consensus.cc:729] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bfd657aa5544d1a0925893d9121c7c, State: Initialized, Role: FOLLOWER
I20240501 08:11:07.634352 10320 consensus_queue.cc:260] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:07.635707 10320 ts_tablet_manager.cc:1398] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c: Time spent starting tablet: real 0.008s user 0.011s sys 0.000s
I20240501 08:11:07.636860 8424 tablet_copy_service.cc:342] P fdc83c70fcc04efda1a05f9d8199b9e9: Request end of tablet copy session 74bfd657aa5544d1a0925893d9121c7c-d71f0f9eeffd476ebd70487ff9a93d7c received from {username='slave'} at 127.6.56.68:55701
I20240501 08:11:07.637043 8424 tablet_copy_service.cc:433] P fdc83c70fcc04efda1a05f9d8199b9e9: ending tablet copy session 74bfd657aa5544d1a0925893d9121c7c-d71f0f9eeffd476ebd70487ff9a93d7c on tablet d71f0f9eeffd476ebd70487ff9a93d7c with peer 74bfd657aa5544d1a0925893d9121c7c
I20240501 08:11:08.000162 9784 raft_consensus.cc:1203] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:08.002010 10329 raft_consensus.cc:2932] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:08.002635 9784 raft_consensus.cc:2932] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:08.006304 10329 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60588 (request call id 0) took 1023 ms. Trace:
I20240501 08:11:08.006525 10329 rpcz_store.cc:276] 0501 08:11:06.982517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:11:06.982577 (+ 60us) service_pool.cc:225] Handling call
0501 08:11:06.984861 (+ 2284us) log.cc:842] Serialized 254 byte log entry
0501 08:11:08.006257 (+1021396us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":73,"raft.queue_time_us":348,"raft.run_cpu_time_us":520,"raft.run_wall_time_us":521,"spinlock_wait_cycles":126336,"thread_start_us":531,"threads_started":2,"wal-append.queue_time_us":379,"wal-append.run_cpu_time_us":534,"wal-append.run_wall_time_us":1000607}
I20240501 08:11:08.008091 8260 catalog_manager.cc:5484] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c reported cstate change: config changed from index -1 to 4, VOTER 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:08.038384 8404 consensus_queue.cc:237] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
W20240501 08:11:08.047217 8338 consensus_peers.cc:485] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 -> Peer ac552fa18d104068846a832248e27bcc (127.6.56.70:33819): Couldn't send request to peer ac552fa18d104068846a832248e27bcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 493df0048a8740f190d7d126e70e8bc5. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:08.574525 10356 ts_tablet_manager.cc:902] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: Initiating tablet copy from peer fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:08.576161 10356 tablet_copy_client.cc:308] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46073
I20240501 08:11:08.583135 8424 tablet_copy_service.cc:140] P fdc83c70fcc04efda1a05f9d8199b9e9: Received BeginTabletCopySession request for tablet 493df0048a8740f190d7d126e70e8bc5 from peer ac552fa18d104068846a832248e27bcc ({username='slave'} at 127.6.56.70:45165)
I20240501 08:11:08.583305 8424 tablet_copy_service.cc:161] P fdc83c70fcc04efda1a05f9d8199b9e9: Beginning new tablet copy session on tablet 493df0048a8740f190d7d126e70e8bc5 from peer ac552fa18d104068846a832248e27bcc at {username='slave'} at 127.6.56.70:45165: session id = ac552fa18d104068846a832248e27bcc-493df0048a8740f190d7d126e70e8bc5
I20240501 08:11:08.585357 8424 tablet_copy_source_session.cc:205] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:08.589964 10356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493df0048a8740f190d7d126e70e8bc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:08.600643 10356 tablet_copy_client.cc:783] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:08.601408 10356 tablet_copy_client.cc:647] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:08.603543 10356 tablet_copy_client.cc:523] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:08.606727 10356 tablet_bootstrap.cc:492] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: Bootstrap starting.
I20240501 08:11:08.615957 10356 log.cc:824] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:08.648392 10356 tablet_bootstrap.cc:492] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:08.649027 10356 tablet_bootstrap.cc:492] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: Bootstrap complete.
I20240501 08:11:08.649469 10356 ts_tablet_manager.cc:1367] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: Time spent bootstrapping tablet: real 0.043s user 0.039s sys 0.004s
I20240501 08:11:08.654633 10356 raft_consensus.cc:348] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:08.655526 10356 raft_consensus.cc:729] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac552fa18d104068846a832248e27bcc, State: Initialized, Role: FOLLOWER
I20240501 08:11:08.656031 10356 consensus_queue.cc:260] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:08.657666 10356 ts_tablet_manager.cc:1398] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240501 08:11:08.658706 8424 tablet_copy_service.cc:342] P fdc83c70fcc04efda1a05f9d8199b9e9: Request end of tablet copy session ac552fa18d104068846a832248e27bcc-493df0048a8740f190d7d126e70e8bc5 received from {username='slave'} at 127.6.56.70:45165
I20240501 08:11:08.658824 8424 tablet_copy_service.cc:433] P fdc83c70fcc04efda1a05f9d8199b9e9: ending tablet copy session ac552fa18d104068846a832248e27bcc-493df0048a8740f190d7d126e70e8bc5 on tablet 493df0048a8740f190d7d126e70e8bc5 with peer ac552fa18d104068846a832248e27bcc
I20240501 08:11:09.073485 10044 raft_consensus.cc:1203] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:09.075186 10365 raft_consensus.cc:2932] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac552fa18d104068846a832248e27bcc (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:09.076061 10044 raft_consensus.cc:2932] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac552fa18d104068846a832248e27bcc (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:09.078157 10365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60594 (request call id 0) took 1040 ms. Trace:
I20240501 08:11:09.078358 10365 rpcz_store.cc:276] 0501 08:11:08.037840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:11:08.037905 (+ 65us) service_pool.cc:225] Handling call
0501 08:11:08.039005 (+ 1100us) log.cc:842] Serialized 254 byte log entry
0501 08:11:09.078105 (+1039100us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":57,"raft.queue_time_us":47,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":189,"spinlock_wait_cycles":1280,"thread_start_us":214,"threads_started":1,"wal-append.queue_time_us":360,"wal-append.run_cpu_time_us":624,"wal-append.run_wall_time_us":1000716}
I20240501 08:11:09.081027 8257 catalog_manager.cc:5484] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: config changed from index -1 to 4, VOTER ac552fa18d104068846a832248e27bcc (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:09.114804 8535 consensus_queue.cc:237] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
W20240501 08:11:09.124801 8468 consensus_peers.cc:485] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 -> Peer e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69:33185): Couldn't send request to peer e88366e322ed40e7bcff964b07cdeaf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0ebe3885c6f4e3794bb61abf33dbb97. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:09.631775 10394 ts_tablet_manager.cc:902] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: Initiating tablet copy from peer 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:09.633430 10394 tablet_copy_client.cc:308] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:40791
I20240501 08:11:09.640198 8555 tablet_copy_service.cc:140] P 2631692a8bb44abfb7d05022e6ae9b94: Received BeginTabletCopySession request for tablet e0ebe3885c6f4e3794bb61abf33dbb97 from peer e88366e322ed40e7bcff964b07cdeaf2 ({username='slave'} at 127.6.56.69:57063)
I20240501 08:11:09.640470 8555 tablet_copy_service.cc:161] P 2631692a8bb44abfb7d05022e6ae9b94: Beginning new tablet copy session on tablet e0ebe3885c6f4e3794bb61abf33dbb97 from peer e88366e322ed40e7bcff964b07cdeaf2 at {username='slave'} at 127.6.56.69:57063: session id = e88366e322ed40e7bcff964b07cdeaf2-e0ebe3885c6f4e3794bb61abf33dbb97
I20240501 08:11:09.643052 8555 tablet_copy_source_session.cc:205] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:09.647841 10394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0ebe3885c6f4e3794bb61abf33dbb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:09.658130 10394 tablet_copy_client.cc:783] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:09.658412 10394 tablet_copy_client.cc:647] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:09.661249 10394 tablet_copy_client.cc:523] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:09.664422 10394 tablet_bootstrap.cc:492] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: Bootstrap starting.
I20240501 08:11:09.673735 10394 log.cc:824] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:09.705601 10394 tablet_bootstrap.cc:492] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:09.706164 10394 tablet_bootstrap.cc:492] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: Bootstrap complete.
I20240501 08:11:09.706607 10394 ts_tablet_manager.cc:1367] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: Time spent bootstrapping tablet: real 0.042s user 0.039s sys 0.004s
I20240501 08:11:09.711629 10394 raft_consensus.cc:348] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:09.712508 10394 raft_consensus.cc:729] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88366e322ed40e7bcff964b07cdeaf2, State: Initialized, Role: FOLLOWER
I20240501 08:11:09.712925 10394 consensus_queue.cc:260] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:09.714319 10394 ts_tablet_manager.cc:1398] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20240501 08:11:09.715377 8555 tablet_copy_service.cc:342] P 2631692a8bb44abfb7d05022e6ae9b94: Request end of tablet copy session e88366e322ed40e7bcff964b07cdeaf2-e0ebe3885c6f4e3794bb61abf33dbb97 received from {username='slave'} at 127.6.56.69:57063
I20240501 08:11:09.715534 8555 tablet_copy_service.cc:433] P 2631692a8bb44abfb7d05022e6ae9b94: ending tablet copy session e88366e322ed40e7bcff964b07cdeaf2-e0ebe3885c6f4e3794bb61abf33dbb97 on tablet e0ebe3885c6f4e3794bb61abf33dbb97 with peer e88366e322ed40e7bcff964b07cdeaf2
I20240501 08:11:10.249735 9914 raft_consensus.cc:1203] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:10.251825 10405 raft_consensus.cc:2932] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:10.252758 9914 raft_consensus.cc:2932] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:10.255123 10405 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32870 (request call id 0) took 1141 ms. Trace:
I20240501 08:11:10.255367 10405 rpcz_store.cc:276] 0501 08:11:09.113747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:11:09.113811 (+ 64us) service_pool.cc:225] Handling call
0501 08:11:09.116001 (+ 2190us) log.cc:842] Serialized 254 byte log entry
0501 08:11:10.255056 (+1139055us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":73,"raft.queue_time_us":803,"raft.run_cpu_time_us":571,"raft.run_wall_time_us":571,"spinlock_wait_cycles":98560,"thread_start_us":1023,"threads_started":2,"wal-append.queue_time_us":413,"wal-append.run_cpu_time_us":1040,"wal-append.run_wall_time_us":1001115}
I20240501 08:11:10.258239 8257 catalog_manager.cc:5484] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 reported cstate change: config changed from index -1 to 4, VOTER e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:10.294039 8535 consensus_queue.cc:237] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
W20240501 08:11:10.302959 8470 consensus_peers.cc:485] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 -> Peer 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71:42449): Couldn't send request to peer 6b0e60a3c4f74fdba1981d86fec4c529. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80636827cb8c427c8317f676d8790c70. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:10.863070 10433 ts_tablet_manager.cc:902] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: Initiating tablet copy from peer 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:10.864854 10433 tablet_copy_client.cc:308] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:40791
I20240501 08:11:10.871104 8555 tablet_copy_service.cc:140] P 2631692a8bb44abfb7d05022e6ae9b94: Received BeginTabletCopySession request for tablet 80636827cb8c427c8317f676d8790c70 from peer 6b0e60a3c4f74fdba1981d86fec4c529 ({username='slave'} at 127.6.56.71:57107)
I20240501 08:11:10.871279 8555 tablet_copy_service.cc:161] P 2631692a8bb44abfb7d05022e6ae9b94: Beginning new tablet copy session on tablet 80636827cb8c427c8317f676d8790c70 from peer 6b0e60a3c4f74fdba1981d86fec4c529 at {username='slave'} at 127.6.56.71:57107: session id = 6b0e60a3c4f74fdba1981d86fec4c529-80636827cb8c427c8317f676d8790c70
I20240501 08:11:10.873277 8555 tablet_copy_source_session.cc:205] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:10.877656 10433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80636827cb8c427c8317f676d8790c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:10.888763 10433 tablet_copy_client.cc:783] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:10.889097 10433 tablet_copy_client.cc:647] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:10.891286 10433 tablet_copy_client.cc:523] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:10.894536 10433 tablet_bootstrap.cc:492] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: Bootstrap starting.
I20240501 08:11:10.903800 10433 log.cc:824] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:10.934290 10433 tablet_bootstrap.cc:492] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:10.934926 10433 tablet_bootstrap.cc:492] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: Bootstrap complete.
I20240501 08:11:10.935374 10433 ts_tablet_manager.cc:1367] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: Time spent bootstrapping tablet: real 0.041s user 0.035s sys 0.004s
I20240501 08:11:10.940433 10433 raft_consensus.cc:348] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:10.941356 10433 raft_consensus.cc:729] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0e60a3c4f74fdba1981d86fec4c529, State: Initialized, Role: FOLLOWER
I20240501 08:11:10.941777 10433 consensus_queue.cc:260] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:10.943639 10433 ts_tablet_manager.cc:1398] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20240501 08:11:10.944835 8555 tablet_copy_service.cc:342] P 2631692a8bb44abfb7d05022e6ae9b94: Request end of tablet copy session 6b0e60a3c4f74fdba1981d86fec4c529-80636827cb8c427c8317f676d8790c70 received from {username='slave'} at 127.6.56.71:57107
I20240501 08:11:10.944981 8555 tablet_copy_service.cc:433] P 2631692a8bb44abfb7d05022e6ae9b94: ending tablet copy session 6b0e60a3c4f74fdba1981d86fec4c529-80636827cb8c427c8317f676d8790c70 on tablet 80636827cb8c427c8317f676d8790c70 with peer 6b0e60a3c4f74fdba1981d86fec4c529
I20240501 08:11:11.334595 10174 raft_consensus.cc:1203] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:11.336371 10442 raft_consensus.cc:2932] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:11.337343 10174 raft_consensus.cc:2932] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:11.339802 10442 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32890 (request call id 0) took 1046 ms. Trace:
I20240501 08:11:11.339989 10442 rpcz_store.cc:276] 0501 08:11:10.293489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:11:10.293557 (+ 68us) service_pool.cc:225] Handling call
0501 08:11:10.294728 (+ 1171us) log.cc:842] Serialized 254 byte log entry
0501 08:11:11.339741 (+1045013us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":70,"raft.queue_time_us":57,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":209,"thread_start_us":266,"threads_started":1,"wal-append.queue_time_us":391,"wal-append.run_cpu_time_us":588,"wal-append.run_wall_time_us":1000652}
I20240501 08:11:11.342204 8260 catalog_manager.cc:5484] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: config changed from index -1 to 4, VOTER 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:11.372778 8535 consensus_queue.cc:237] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
W20240501 08:11:11.381250 8469 consensus_peers.cc:485] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 -> Peer 0ac549325eca4b79be5c3861c96f23b2 (127.6.56.67:42103): Couldn't send request to peer 0ac549325eca4b79be5c3861c96f23b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f6207df29954aa3bfbc81304bd38827. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:11.844560 10468 ts_tablet_manager.cc:902] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: Initiating tablet copy from peer 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:11.846227 10468 tablet_copy_client.cc:308] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:40791
I20240501 08:11:11.853278 8555 tablet_copy_service.cc:140] P 2631692a8bb44abfb7d05022e6ae9b94: Received BeginTabletCopySession request for tablet 1f6207df29954aa3bfbc81304bd38827 from peer 0ac549325eca4b79be5c3861c96f23b2 ({username='slave'} at 127.6.56.67:38645)
I20240501 08:11:11.853451 8555 tablet_copy_service.cc:161] P 2631692a8bb44abfb7d05022e6ae9b94: Beginning new tablet copy session on tablet 1f6207df29954aa3bfbc81304bd38827 from peer 0ac549325eca4b79be5c3861c96f23b2 at {username='slave'} at 127.6.56.67:38645: session id = 0ac549325eca4b79be5c3861c96f23b2-1f6207df29954aa3bfbc81304bd38827
I20240501 08:11:11.855358 8555 tablet_copy_source_session.cc:205] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:11.857579 10468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f6207df29954aa3bfbc81304bd38827. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:11.863420 10468 tablet_copy_client.cc:783] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:11.863725 10468 tablet_copy_client.cc:647] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:11.865922 10468 tablet_copy_client.cc:523] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:11.868636 10468 tablet_bootstrap.cc:492] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: Bootstrap starting.
I20240501 08:11:11.902930 10468 tablet_bootstrap.cc:492] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:11.903461 10468 tablet_bootstrap.cc:492] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: Bootstrap complete.
I20240501 08:11:11.903733 10468 ts_tablet_manager.cc:1367] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.007s
I20240501 08:11:11.904593 10468 raft_consensus.cc:348] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:11.905068 10468 raft_consensus.cc:729] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac549325eca4b79be5c3861c96f23b2, State: Initialized, Role: FOLLOWER
I20240501 08:11:11.905254 10468 consensus_queue.cc:260] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:11:11.905925 10468 ts_tablet_manager.cc:1398] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 08:11:11.906855 8555 tablet_copy_service.cc:342] P 2631692a8bb44abfb7d05022e6ae9b94: Request end of tablet copy session 0ac549325eca4b79be5c3861c96f23b2-1f6207df29954aa3bfbc81304bd38827 received from {username='slave'} at 127.6.56.67:38645
I20240501 08:11:11.907002 8555 tablet_copy_service.cc:433] P 2631692a8bb44abfb7d05022e6ae9b94: ending tablet copy session 0ac549325eca4b79be5c3861c96f23b2-1f6207df29954aa3bfbc81304bd38827 on tablet 1f6207df29954aa3bfbc81304bd38827 with peer 0ac549325eca4b79be5c3861c96f23b2
I20240501 08:11:12.381600 9644 raft_consensus.cc:1203] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:12.383188 10476 raft_consensus.cc:2932] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ac549325eca4b79be5c3861c96f23b2 (127.6.56.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } } }
I20240501 08:11:12.383898 9644 raft_consensus.cc:2932] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 0ac549325eca4b79be5c3861c96f23b2 (127.6.56.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } } }
I20240501 08:11:12.386015 10476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32912 (request call id 0) took 1013 ms. Trace:
I20240501 08:11:12.386229 10476 rpcz_store.cc:276] 0501 08:11:11.372163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:11:11.372233 (+ 70us) service_pool.cc:225] Handling call
0501 08:11:11.373470 (+ 1237us) log.cc:842] Serialized 254 byte log entry
0501 08:11:12.385963 (+1012493us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":68,"raft.queue_time_us":42,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":249,"spinlock_wait_cycles":69888,"thread_start_us":197,"threads_started":1,"wal-append.queue_time_us":291,"wal-append.run_cpu_time_us":587,"wal-append.run_wall_time_us":1000620}
I20240501 08:11:12.388458 8260 catalog_manager.cc:5484] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: config changed from index -1 to 4, VOTER 0ac549325eca4b79be5c3861c96f23b2 (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } } }
I20240501 08:11:12.423599 8535 consensus_queue.cc:237] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
W20240501 08:11:12.430732 8469 consensus_peers.cc:485] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 -> Peer 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68:38709): Couldn't send request to peer 74bfd657aa5544d1a0925893d9121c7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 833a0a7dc6a047b19808085569e89368. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:12.935993 10503 ts_tablet_manager.cc:902] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: Initiating tablet copy from peer 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:12.937165 10503 tablet_copy_client.cc:308] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:40791
I20240501 08:11:12.943580 8555 tablet_copy_service.cc:140] P 2631692a8bb44abfb7d05022e6ae9b94: Received BeginTabletCopySession request for tablet 833a0a7dc6a047b19808085569e89368 from peer 74bfd657aa5544d1a0925893d9121c7c ({username='slave'} at 127.6.56.68:53677)
I20240501 08:11:12.943734 8555 tablet_copy_service.cc:161] P 2631692a8bb44abfb7d05022e6ae9b94: Beginning new tablet copy session on tablet 833a0a7dc6a047b19808085569e89368 from peer 74bfd657aa5544d1a0925893d9121c7c at {username='slave'} at 127.6.56.68:53677: session id = 74bfd657aa5544d1a0925893d9121c7c-833a0a7dc6a047b19808085569e89368
I20240501 08:11:12.945740 8555 tablet_copy_source_session.cc:205] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:12.946885 10503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833a0a7dc6a047b19808085569e89368. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:12.951223 10503 tablet_copy_client.cc:783] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:12.951478 10503 tablet_copy_client.cc:647] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:12.953168 10503 tablet_copy_client.cc:523] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:12.955451 10503 tablet_bootstrap.cc:492] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: Bootstrap starting.
I20240501 08:11:12.986631 10503 tablet_bootstrap.cc:492] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:12.987010 10503 tablet_bootstrap.cc:492] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: Bootstrap complete.
I20240501 08:11:12.987236 10503 ts_tablet_manager.cc:1367] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: Time spent bootstrapping tablet: real 0.032s user 0.031s sys 0.000s
I20240501 08:11:12.988132 10503 raft_consensus.cc:348] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:12.988499 10503 raft_consensus.cc:729] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bfd657aa5544d1a0925893d9121c7c, State: Initialized, Role: FOLLOWER
I20240501 08:11:12.988646 10503 consensus_queue.cc:260] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:12.989333 10503 ts_tablet_manager.cc:1398] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 08:11:12.990237 8555 tablet_copy_service.cc:342] P 2631692a8bb44abfb7d05022e6ae9b94: Request end of tablet copy session 74bfd657aa5544d1a0925893d9121c7c-833a0a7dc6a047b19808085569e89368 received from {username='slave'} at 127.6.56.68:53677
I20240501 08:11:12.990391 8555 tablet_copy_service.cc:433] P 2631692a8bb44abfb7d05022e6ae9b94: ending tablet copy session 74bfd657aa5544d1a0925893d9121c7c-833a0a7dc6a047b19808085569e89368 on tablet 833a0a7dc6a047b19808085569e89368 with peer 74bfd657aa5544d1a0925893d9121c7c
I20240501 08:11:13.435778 9784 raft_consensus.cc:1203] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:13.437062 10512 raft_consensus.cc:2932] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:13.437749 9784 raft_consensus.cc:2932] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:13.440079 10512 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:32930 (request call id 0) took 1017 ms. Trace:
I20240501 08:11:13.440254 10512 rpcz_store.cc:276] 0501 08:11:12.423055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:11:12.423125 (+ 70us) service_pool.cc:225] Handling call
0501 08:11:12.424216 (+ 1091us) log.cc:842] Serialized 254 byte log entry
0501 08:11:13.440027 (+1015811us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":73,"raft.queue_time_us":53,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":216,"thread_start_us":240,"threads_started":1,"wal-append.queue_time_us":383,"wal-append.run_cpu_time_us":553,"wal-append.run_wall_time_us":1000628}
I20240501 08:11:13.442881 8256 catalog_manager.cc:5484] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 reported cstate change: config changed from index -1 to 4, VOTER 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:13.478022 8404 consensus_queue.cc:237] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
W20240501 08:11:13.480047 8338 consensus_peers.cc:485] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 -> Peer ac552fa18d104068846a832248e27bcc (127.6.56.70:33819): Couldn't send request to peer ac552fa18d104068846a832248e27bcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d227b998061e46f59653167740f87fd0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:13.936442 10537 ts_tablet_manager.cc:902] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: Initiating tablet copy from peer fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:13.937558 10537 tablet_copy_client.cc:308] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46073
I20240501 08:11:13.938313 8424 tablet_copy_service.cc:140] P fdc83c70fcc04efda1a05f9d8199b9e9: Received BeginTabletCopySession request for tablet d227b998061e46f59653167740f87fd0 from peer ac552fa18d104068846a832248e27bcc ({username='slave'} at 127.6.56.70:45165)
I20240501 08:11:13.938462 8424 tablet_copy_service.cc:161] P fdc83c70fcc04efda1a05f9d8199b9e9: Beginning new tablet copy session on tablet d227b998061e46f59653167740f87fd0 from peer ac552fa18d104068846a832248e27bcc at {username='slave'} at 127.6.56.70:45165: session id = ac552fa18d104068846a832248e27bcc-d227b998061e46f59653167740f87fd0
I20240501 08:11:13.940500 8424 tablet_copy_source_session.cc:205] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:13.941699 10537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d227b998061e46f59653167740f87fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:13.946626 10537 tablet_copy_client.cc:783] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:13.946904 10537 tablet_copy_client.cc:647] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:13.948495 10537 tablet_copy_client.cc:523] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:13.950949 10537 tablet_bootstrap.cc:492] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: Bootstrap starting.
I20240501 08:11:13.982986 10537 tablet_bootstrap.cc:492] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:13.983402 10537 tablet_bootstrap.cc:492] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: Bootstrap complete.
I20240501 08:11:13.983646 10537 ts_tablet_manager.cc:1367] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: Time spent bootstrapping tablet: real 0.033s user 0.028s sys 0.004s
I20240501 08:11:13.984557 10537 raft_consensus.cc:348] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:13.984948 10537 raft_consensus.cc:729] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac552fa18d104068846a832248e27bcc, State: Initialized, Role: FOLLOWER
I20240501 08:11:13.985127 10537 consensus_queue.cc:260] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:13.985785 10537 ts_tablet_manager.cc:1398] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 08:11:13.986708 8424 tablet_copy_service.cc:342] P fdc83c70fcc04efda1a05f9d8199b9e9: Request end of tablet copy session ac552fa18d104068846a832248e27bcc-d227b998061e46f59653167740f87fd0 received from {username='slave'} at 127.6.56.70:45165
I20240501 08:11:13.986876 8424 tablet_copy_service.cc:433] P fdc83c70fcc04efda1a05f9d8199b9e9: ending tablet copy session ac552fa18d104068846a832248e27bcc-d227b998061e46f59653167740f87fd0 on tablet d227b998061e46f59653167740f87fd0 with peer ac552fa18d104068846a832248e27bcc
I20240501 08:11:14.449075 10044 raft_consensus.cc:1203] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:14.450474 10544 raft_consensus.cc:2932] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac552fa18d104068846a832248e27bcc (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:14.452519 10044 raft_consensus.cc:2932] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER ac552fa18d104068846a832248e27bcc (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:14.455674 8257 catalog_manager.cc:5484] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: config changed from index -1 to 4, VOTER ac552fa18d104068846a832248e27bcc (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:14.498415 8404 consensus_queue.cc:237] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
W20240501 08:11:14.508764 8337 consensus_peers.cc:485] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 -> Peer e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69:33185): Couldn't send request to peer e88366e322ed40e7bcff964b07cdeaf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 532f21629fae48408e569bdcd04c2cbd. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:14.938791 10570 ts_tablet_manager.cc:902] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: Initiating tablet copy from peer fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:14.940095 10570 tablet_copy_client.cc:308] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46073
I20240501 08:11:14.950394 8424 tablet_copy_service.cc:140] P fdc83c70fcc04efda1a05f9d8199b9e9: Received BeginTabletCopySession request for tablet 532f21629fae48408e569bdcd04c2cbd from peer e88366e322ed40e7bcff964b07cdeaf2 ({username='slave'} at 127.6.56.69:42789)
I20240501 08:11:14.950598 8424 tablet_copy_service.cc:161] P fdc83c70fcc04efda1a05f9d8199b9e9: Beginning new tablet copy session on tablet 532f21629fae48408e569bdcd04c2cbd from peer e88366e322ed40e7bcff964b07cdeaf2 at {username='slave'} at 127.6.56.69:42789: session id = e88366e322ed40e7bcff964b07cdeaf2-532f21629fae48408e569bdcd04c2cbd
I20240501 08:11:14.952639 8424 tablet_copy_source_session.cc:205] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:14.954011 10570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 532f21629fae48408e569bdcd04c2cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:14.958513 10570 tablet_copy_client.cc:783] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:14.958750 10570 tablet_copy_client.cc:647] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:14.960666 10570 tablet_copy_client.cc:523] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:14.963227 10570 tablet_bootstrap.cc:492] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: Bootstrap starting.
I20240501 08:11:14.993781 10570 tablet_bootstrap.cc:492] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:14.994249 10570 tablet_bootstrap.cc:492] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: Bootstrap complete.
I20240501 08:11:14.994508 10570 ts_tablet_manager.cc:1367] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.004s
I20240501 08:11:14.995318 10570 raft_consensus.cc:348] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:14.995702 10570 raft_consensus.cc:729] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e88366e322ed40e7bcff964b07cdeaf2, State: Initialized, Role: FOLLOWER
I20240501 08:11:14.995877 10570 consensus_queue.cc:260] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:14.996636 10570 ts_tablet_manager.cc:1398] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240501 08:11:14.997480 8424 tablet_copy_service.cc:342] P fdc83c70fcc04efda1a05f9d8199b9e9: Request end of tablet copy session e88366e322ed40e7bcff964b07cdeaf2-532f21629fae48408e569bdcd04c2cbd received from {username='slave'} at 127.6.56.69:42789
I20240501 08:11:14.997603 8424 tablet_copy_service.cc:433] P fdc83c70fcc04efda1a05f9d8199b9e9: ending tablet copy session e88366e322ed40e7bcff964b07cdeaf2-532f21629fae48408e569bdcd04c2cbd on tablet 532f21629fae48408e569bdcd04c2cbd with peer e88366e322ed40e7bcff964b07cdeaf2
I20240501 08:11:15.446125 9914 raft_consensus.cc:1203] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:15.448060 10578 raft_consensus.cc:2932] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:15.448828 9914 raft_consensus.cc:2932] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:15.455020 8260 catalog_manager.cc:5484] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: config changed from index -1 to 4, VOTER e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:15.495648 8404 consensus_queue.cc:237] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
W20240501 08:11:15.503697 8339 consensus_peers.cc:485] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 -> Peer 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71:42449): Couldn't send request to peer 6b0e60a3c4f74fdba1981d86fec4c529. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9cfe3dfff4848aabb0c7b420717130c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:15.983645 10603 ts_tablet_manager.cc:902] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: Initiating tablet copy from peer fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:15.985035 10603 tablet_copy_client.cc:308] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46073
I20240501 08:11:15.991132 8424 tablet_copy_service.cc:140] P fdc83c70fcc04efda1a05f9d8199b9e9: Received BeginTabletCopySession request for tablet e9cfe3dfff4848aabb0c7b420717130c from peer 6b0e60a3c4f74fdba1981d86fec4c529 ({username='slave'} at 127.6.56.71:38321)
I20240501 08:11:15.991314 8424 tablet_copy_service.cc:161] P fdc83c70fcc04efda1a05f9d8199b9e9: Beginning new tablet copy session on tablet e9cfe3dfff4848aabb0c7b420717130c from peer 6b0e60a3c4f74fdba1981d86fec4c529 at {username='slave'} at 127.6.56.71:38321: session id = 6b0e60a3c4f74fdba1981d86fec4c529-e9cfe3dfff4848aabb0c7b420717130c
I20240501 08:11:15.993724 8424 tablet_copy_source_session.cc:205] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:15.994993 10603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9cfe3dfff4848aabb0c7b420717130c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:15.999441 10603 tablet_copy_client.cc:783] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:15.999701 10603 tablet_copy_client.cc:647] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:16.001525 10603 tablet_copy_client.cc:523] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:16.004062 10603 tablet_bootstrap.cc:492] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: Bootstrap starting.
I20240501 08:11:16.035457 10603 tablet_bootstrap.cc:492] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:16.035843 10603 tablet_bootstrap.cc:492] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: Bootstrap complete.
I20240501 08:11:16.036072 10603 ts_tablet_manager.cc:1367] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20240501 08:11:16.037021 10603 raft_consensus.cc:348] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:16.037387 10603 raft_consensus.cc:729] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b0e60a3c4f74fdba1981d86fec4c529, State: Initialized, Role: FOLLOWER
I20240501 08:11:16.037542 10603 consensus_queue.cc:260] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:16.038688 10603 ts_tablet_manager.cc:1398] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20240501 08:11:16.039556 8424 tablet_copy_service.cc:342] P fdc83c70fcc04efda1a05f9d8199b9e9: Request end of tablet copy session 6b0e60a3c4f74fdba1981d86fec4c529-e9cfe3dfff4848aabb0c7b420717130c received from {username='slave'} at 127.6.56.71:38321
I20240501 08:11:16.039698 8424 tablet_copy_service.cc:433] P fdc83c70fcc04efda1a05f9d8199b9e9: ending tablet copy session 6b0e60a3c4f74fdba1981d86fec4c529-e9cfe3dfff4848aabb0c7b420717130c on tablet e9cfe3dfff4848aabb0c7b420717130c with peer 6b0e60a3c4f74fdba1981d86fec4c529
I20240501 08:11:16.378388 10174 raft_consensus.cc:1203] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:16.379657 10611 raft_consensus.cc:2932] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:16.380247 10174 raft_consensus.cc:2932] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:16.384892 8256 catalog_manager.cc:5484] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: config changed from index -1 to 4, VOTER 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:16.508188 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:16.511875 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:16.513756 10154 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:16.552482 9764 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:16.592941 8383 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:16.606906 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:16.609351 10024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:16.661010 8404 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d227b998061e46f59653167740f87fd0"
dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35204
I20240501 08:11:16.661361 8404 raft_consensus.cc:595] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:16.757850 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:16.772939 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:16.774974 10154 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:16.777161 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:16.779131 9764 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:16.780078 10024 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:16.807659 8383 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:11:16.963513 10729 raft_consensus.cc:979] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: : Instructing follower ac552fa18d104068846a832248e27bcc to start an election
I20240501 08:11:16.966382 10611 raft_consensus.cc:1067] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Signalling peer ac552fa18d104068846a832248e27bcc to start an election
I20240501 08:11:16.967437 10044 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d227b998061e46f59653167740f87fd0"
dest_uuid: "ac552fa18d104068846a832248e27bcc"
from {username='slave'} at 127.6.56.65:60311
I20240501 08:11:16.967759 10044 raft_consensus.cc:482] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:16.967958 10044 raft_consensus.cc:3037] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:16.970321 10044 raft_consensus.cc:504] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:16.971601 10044 leader_election.cc:288] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [CANDIDATE]: Term 2 election: Requested vote from peers fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:16.972270 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d227b998061e46f59653167740f87fd0" candidate_uuid: "ac552fa18d104068846a832248e27bcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
I20240501 08:11:16.972626 8404 raft_consensus.cc:3032] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:16.972726 8404 raft_consensus.cc:729] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:16.973091 8404 consensus_queue.cc:260] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:16.973369 8404 raft_consensus.cc:3037] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:16.975769 8404 raft_consensus.cc:2454] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac552fa18d104068846a832248e27bcc in term 2.
I20240501 08:11:16.976527 9977 leader_election.cc:302] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac552fa18d104068846a832248e27bcc, fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:16.977807 10733 raft_consensus.cc:2781] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:16.978920 10733 raft_consensus.cc:686] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 2 LEADER]: Becoming Leader. State: Replica: ac552fa18d104068846a832248e27bcc, State: Running, Role: LEADER
I20240501 08:11:16.979943 10733 consensus_queue.cc:237] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:16.983222 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:16.985170 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:16.996574 8260 catalog_manager.cc:5484] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc reported cstate change: term changed from 1 to 2, leader changed from fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) to ac552fa18d104068846a832248e27bcc (127.6.56.70). New cstate: current_term: 2 leader_uuid: "ac552fa18d104068846a832248e27bcc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:17.001178 10154 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.014245 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.018035 9764 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.032213 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.097682 8383 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:17.176751 8404 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d71f0f9eeffd476ebd70487ff9a93d7c"
dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35242
I20240501 08:11:17.176978 8404 raft_consensus.cc:595] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:17.269728 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.281592 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:17.289927 9764 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.290271 10154 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.301767 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.312000 8383 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:17.335261 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.388937 10611 raft_consensus.cc:979] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: : Instructing follower 74bfd657aa5544d1a0925893d9121c7c to start an election
I20240501 08:11:17.389122 10611 raft_consensus.cc:1067] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Signalling peer 74bfd657aa5544d1a0925893d9121c7c to start an election
I20240501 08:11:17.390020 9784 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d71f0f9eeffd476ebd70487ff9a93d7c"
dest_uuid: "74bfd657aa5544d1a0925893d9121c7c"
from {username='slave'} at 127.6.56.65:53623
I20240501 08:11:17.390367 9784 raft_consensus.cc:482] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:17.390590 9784 raft_consensus.cc:3037] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:17.393736 9784 raft_consensus.cc:504] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:17.395144 9784 leader_election.cc:288] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [CANDIDATE]: Term 2 election: Requested vote from peers fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:17.395618 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71f0f9eeffd476ebd70487ff9a93d7c" candidate_uuid: "74bfd657aa5544d1a0925893d9121c7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
I20240501 08:11:17.395854 8404 raft_consensus.cc:3032] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:17.395951 8404 raft_consensus.cc:729] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:17.396224 8404 consensus_queue.cc:260] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:17.396602 8404 raft_consensus.cc:3037] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:17.398375 8404 raft_consensus.cc:2454] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74bfd657aa5544d1a0925893d9121c7c in term 2.
I20240501 08:11:17.398875 9717 leader_election.cc:302] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 74bfd657aa5544d1a0925893d9121c7c, fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:17.399657 10783 raft_consensus.cc:2781] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:17.400175 10783 raft_consensus.cc:686] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 2 LEADER]: Becoming Leader. State: Replica: 74bfd657aa5544d1a0925893d9121c7c, State: Running, Role: LEADER
I20240501 08:11:17.400758 10783 consensus_queue.cc:237] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:17.405546 8261 catalog_manager.cc:5484] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c reported cstate change: term changed from 1 to 2, leader changed from fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) to 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68). New cstate: current_term: 2 leader_uuid: "74bfd657aa5544d1a0925893d9121c7c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:17.493443 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.505002 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:17.507587 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.527263 10154 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.527441 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.539196 8383 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:17.552006 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.601866 8404 raft_consensus.cc:1261] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Refusing update from remote peer ac552fa18d104068846a832248e27bcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:17.602811 10818 consensus_queue.cc:1035] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:17.640079 8404 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9cfe3dfff4848aabb0c7b420717130c"
dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35278
I20240501 08:11:17.640286 8404 raft_consensus.cc:595] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:17.739365 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.741938 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.748869 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:17.763684 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.767769 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.769222 10154 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:17.783192 8383 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:17.819957 10611 raft_consensus.cc:979] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: : Instructing follower 6b0e60a3c4f74fdba1981d86fec4c529 to start an election
I20240501 08:11:17.820127 10611 raft_consensus.cc:1067] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Signalling peer 6b0e60a3c4f74fdba1981d86fec4c529 to start an election
I20240501 08:11:17.821005 10174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9cfe3dfff4848aabb0c7b420717130c"
dest_uuid: "6b0e60a3c4f74fdba1981d86fec4c529"
from {username='slave'} at 127.6.56.65:47903
I20240501 08:11:17.821308 10174 raft_consensus.cc:482] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:17.821471 10174 raft_consensus.cc:3037] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:17.824760 10174 raft_consensus.cc:504] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:17.827638 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9cfe3dfff4848aabb0c7b420717130c" candidate_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
I20240501 08:11:17.827896 8404 raft_consensus.cc:3032] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:17.827996 8404 raft_consensus.cc:729] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:17.828261 8404 consensus_queue.cc:260] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:17.828517 8404 raft_consensus.cc:3037] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:17.830304 10174 leader_election.cc:288] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [CANDIDATE]: Term 2 election: Requested vote from peers fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:17.831523 8404 raft_consensus.cc:2454] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b0e60a3c4f74fdba1981d86fec4c529 in term 2.
I20240501 08:11:17.832798 10107 leader_election.cc:302] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6b0e60a3c4f74fdba1981d86fec4c529, fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:17.834686 10859 raft_consensus.cc:2781] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:17.835255 10859 raft_consensus.cc:686] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 2 LEADER]: Becoming Leader. State: Replica: 6b0e60a3c4f74fdba1981d86fec4c529, State: Running, Role: LEADER
I20240501 08:11:17.836441 10859 consensus_queue.cc:237] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:17.841058 8255 catalog_manager.cc:5484] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 reported cstate change: term changed from 1 to 2, leader changed from fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) to 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71). New cstate: current_term: 2 leader_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:17.950569 8404 raft_consensus.cc:1261] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Refusing update from remote peer 74bfd657aa5544d1a0925893d9121c7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:17.951684 10899 consensus_queue.cc:1035] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:17.974076 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.983747 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:17.985642 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.990484 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:17.995973 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:18.031468 8383 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:18.032395 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.099037 8535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e0ebe3885c6f4e3794bb61abf33dbb97"
dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42116
I20240501 08:11:18.099326 8535 raft_consensus.cc:595] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:18.214040 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.214813 8515 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:11:18.217526 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.225551 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.237540 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:18.246148 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.248883 8383 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:18.293036 8404 raft_consensus.cc:1261] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Refusing update from remote peer 6b0e60a3c4f74fdba1981d86fec4c529: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:18.293812 10859 consensus_queue.cc:1035] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:18.420890 10978 raft_consensus.cc:979] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: : Instructing follower e88366e322ed40e7bcff964b07cdeaf2 to start an election
I20240501 08:11:18.421190 10476 raft_consensus.cc:1067] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Signalling peer e88366e322ed40e7bcff964b07cdeaf2 to start an election
I20240501 08:11:18.424053 9914 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e0ebe3885c6f4e3794bb61abf33dbb97"
dest_uuid: "e88366e322ed40e7bcff964b07cdeaf2"
from {username='slave'} at 127.6.56.66:39987
I20240501 08:11:18.424468 9914 raft_consensus.cc:482] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:18.424713 9914 raft_consensus.cc:3037] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:18.429147 9914 raft_consensus.cc:504] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:18.432157 9914 leader_election.cc:288] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [CANDIDATE]: Term 2 election: Requested vote from peers 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:18.432332 8535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0ebe3885c6f4e3794bb61abf33dbb97" candidate_uuid: "e88366e322ed40e7bcff964b07cdeaf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
I20240501 08:11:18.432725 8535 raft_consensus.cc:3032] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:18.432855 8535 raft_consensus.cc:729] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:18.433238 8535 consensus_queue.cc:260] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:18.433866 8535 raft_consensus.cc:3037] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:18.437862 9623 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.439924 8535 raft_consensus.cc:2454] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e88366e322ed40e7bcff964b07cdeaf2 in term 2.
I20240501 08:11:18.442063 9849 leader_election.cc:302] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94, e88366e322ed40e7bcff964b07cdeaf2; no voters:
I20240501 08:11:18.451114 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.462666 8515 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:11:18.465940 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.472427 9894 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:18.484726 8383 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:18.486609 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.499771 10988 raft_consensus.cc:2781] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 08:11:18.500525 9849 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.66:40791, user_credentials={real_user=slave}} blocked reactor thread for 58994.3us
I20240501 08:11:18.516561 10988 raft_consensus.cc:686] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 2 LEADER]: Becoming Leader. State: Replica: e88366e322ed40e7bcff964b07cdeaf2, State: Running, Role: LEADER
I20240501 08:11:18.517382 10988 consensus_queue.cc:237] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:18.521631 8260 catalog_manager.cc:5484] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 reported cstate change: term changed from 1 to 2, leader changed from 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) to e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69). New cstate: current_term: 2 leader_uuid: "e88366e322ed40e7bcff964b07cdeaf2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:18.602311 8535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1f6207df29954aa3bfbc81304bd38827"
dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42154
I20240501 08:11:18.602514 8535 raft_consensus.cc:595] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:18.611419 10476 raft_consensus.cc:979] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: : Instructing follower 0ac549325eca4b79be5c3861c96f23b2 to start an election
I20240501 08:11:18.611572 10476 raft_consensus.cc:1067] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Signalling peer 0ac549325eca4b79be5c3861c96f23b2 to start an election
I20240501 08:11:18.612272 9644 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1f6207df29954aa3bfbc81304bd38827"
dest_uuid: "0ac549325eca4b79be5c3861c96f23b2"
from {username='slave'} at 127.6.56.66:51639
I20240501 08:11:18.612607 9644 raft_consensus.cc:482] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:18.612751 9644 raft_consensus.cc:3037] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:18.615829 9644 raft_consensus.cc:504] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:11:18.617163 9644 leader_election.cc:288] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [CANDIDATE]: Term 2 election: Requested vote from peers 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:18.617688 8535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f6207df29954aa3bfbc81304bd38827" candidate_uuid: "0ac549325eca4b79be5c3861c96f23b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
I20240501 08:11:18.617918 8535 raft_consensus.cc:3032] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:18.618012 8535 raft_consensus.cc:729] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:18.618230 8535 consensus_queue.cc:260] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:11:18.618468 8535 raft_consensus.cc:3037] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:18.620213 8535 raft_consensus.cc:2454] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ac549325eca4b79be5c3861c96f23b2 in term 2.
I20240501 08:11:18.620720 9577 leader_election.cc:302] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0ac549325eca4b79be5c3861c96f23b2, 2631692a8bb44abfb7d05022e6ae9b94; no voters:
I20240501 08:11:18.621325 11007 raft_consensus.cc:2781] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:18.621742 11007 raft_consensus.cc:686] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 2 LEADER]: Becoming Leader. State: Replica: 0ac549325eca4b79be5c3861c96f23b2, State: Running, Role: LEADER
I20240501 08:11:18.622069 11007 consensus_queue.cc:237] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:11:18.625650 8260 catalog_manager.cc:5484] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 reported cstate change: term changed from 1 to 2, leader changed from 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) to 0ac549325eca4b79be5c3861c96f23b2 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "0ac549325eca4b79be5c3861c96f23b2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } } }
I20240501 08:11:18.701985 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:18.704561 8515 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:18.715126 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.724647 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.745419 9894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.754407 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.754536 8383 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:18.906525 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:18.911393 8515 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:18.927076 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.927109 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.937762 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.964779 9894 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:18.983014 8383 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:19.052336 8535 raft_consensus.cc:1261] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Refusing update from remote peer e88366e322ed40e7bcff964b07cdeaf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:19.053225 11078 consensus_queue.cc:1035] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:19.053808 8404 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "532f21629fae48408e569bdcd04c2cbd"
dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35356
I20240501 08:11:19.053987 8404 raft_consensus.cc:595] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:19.096963 8535 raft_consensus.cc:1261] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Refusing update from remote peer 0ac549325eca4b79be5c3861c96f23b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:19.097707 11007 consensus_queue.cc:1035] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:19.167011 11116 raft_consensus.cc:979] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: : Instructing follower e88366e322ed40e7bcff964b07cdeaf2 to start an election
I20240501 08:11:19.167325 10729 raft_consensus.cc:1067] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Signalling peer e88366e322ed40e7bcff964b07cdeaf2 to start an election
I20240501 08:11:19.169013 9914 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "532f21629fae48408e569bdcd04c2cbd"
dest_uuid: "e88366e322ed40e7bcff964b07cdeaf2"
from {username='slave'} at 127.6.56.65:52607
I20240501 08:11:19.169309 9914 raft_consensus.cc:482] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:19.169435 9914 raft_consensus.cc:3037] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:19.173975 9914 raft_consensus.cc:504] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:19.175295 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "532f21629fae48408e569bdcd04c2cbd" candidate_uuid: "e88366e322ed40e7bcff964b07cdeaf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
I20240501 08:11:19.175508 8404 raft_consensus.cc:3032] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:19.175596 8404 raft_consensus.cc:729] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:19.175843 8404 consensus_queue.cc:260] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:19.176086 8404 raft_consensus.cc:3037] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:19.177747 8515 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:19.178946 8404 raft_consensus.cc:2454] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e88366e322ed40e7bcff964b07cdeaf2 in term 2.
I20240501 08:11:19.179665 9847 leader_election.cc:302] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: e88366e322ed40e7bcff964b07cdeaf2, fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:19.180166 11084 raft_consensus.cc:2781] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:19.180348 9914 leader_election.cc:288] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [CANDIDATE]: Term 2 election: Requested vote from peers fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:19.180379 11084 raft_consensus.cc:686] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 2 LEADER]: Becoming Leader. State: Replica: e88366e322ed40e7bcff964b07cdeaf2, State: Running, Role: LEADER
I20240501 08:11:19.180724 11084 consensus_queue.cc:237] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:19.189169 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.190784 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.193339 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.193828 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.208884 8383 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:19.246202 8261 catalog_manager.cc:5484] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 reported cstate change: term changed from 1 to 2, leader changed from fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) to e88366e322ed40e7bcff964b07cdeaf2 (127.6.56.69). New cstate: current_term: 2 leader_uuid: "e88366e322ed40e7bcff964b07cdeaf2" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:19.248154 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.409785 8515 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:19.416730 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.420360 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.420480 10024 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.430100 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.436861 8383 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:19.449106 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.535192 8404 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "493df0048a8740f190d7d126e70e8bc5"
dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35380
I20240501 08:11:19.535481 8404 raft_consensus.cc:595] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:19.546753 11116 raft_consensus.cc:979] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: : Instructing follower ac552fa18d104068846a832248e27bcc to start an election
I20240501 08:11:19.546927 11116 raft_consensus.cc:1067] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Signalling peer ac552fa18d104068846a832248e27bcc to start an election
I20240501 08:11:19.547825 10044 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "493df0048a8740f190d7d126e70e8bc5"
dest_uuid: "ac552fa18d104068846a832248e27bcc"
from {username='slave'} at 127.6.56.65:60311
I20240501 08:11:19.548122 10044 raft_consensus.cc:482] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:19.548265 10044 raft_consensus.cc:3037] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:19.552126 10044 raft_consensus.cc:504] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:19.553344 10044 leader_election.cc:288] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [CANDIDATE]: Term 2 election: Requested vote from peers fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:19.560667 8404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "493df0048a8740f190d7d126e70e8bc5" candidate_uuid: "ac552fa18d104068846a832248e27bcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
I20240501 08:11:19.560954 8404 raft_consensus.cc:3032] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:19.561085 8404 raft_consensus.cc:729] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:19.561385 8404 consensus_queue.cc:260] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:19.561774 8404 raft_consensus.cc:3037] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:19.571908 8404 raft_consensus.cc:2454] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac552fa18d104068846a832248e27bcc in term 2.
I20240501 08:11:19.572705 9977 leader_election.cc:302] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac552fa18d104068846a832248e27bcc, fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:19.573462 11006 raft_consensus.cc:2781] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:19.593118 11006 raft_consensus.cc:686] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 2 LEADER]: Becoming Leader. State: Replica: ac552fa18d104068846a832248e27bcc, State: Running, Role: LEADER
I20240501 08:11:19.593632 11006 consensus_queue.cc:237] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:19.607260 8255 catalog_manager.cc:5484] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc reported cstate change: term changed from 1 to 2, leader changed from fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) to ac552fa18d104068846a832248e27bcc (127.6.56.70). New cstate: current_term: 2 leader_uuid: "ac552fa18d104068846a832248e27bcc" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:19.729316 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.742053 8515 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:19.744151 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.751371 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.757102 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:19.773170 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.780985 8404 raft_consensus.cc:1261] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Refusing update from remote peer e88366e322ed40e7bcff964b07cdeaf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:19.781723 11084 consensus_queue.cc:1035] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:19.803133 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:19.974445 8515 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:19.979686 8403 raft_consensus.cc:1261] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Refusing update from remote peer ac552fa18d104068846a832248e27bcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:19.980688 11006 consensus_queue.cc:1035] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:19.987947 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.989399 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:19.999245 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.006457 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.031924 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.052763 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:20.108815 8535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "80636827cb8c427c8317f676d8790c70"
dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42244
I20240501 08:11:20.109006 8535 raft_consensus.cc:595] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:20.203895 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.215466 10154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:20.225543 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:20.235975 8515 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:11:20.251937 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.255668 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.267426 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:20.331372 11279 raft_consensus.cc:979] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: : Instructing follower 6b0e60a3c4f74fdba1981d86fec4c529 to start an election
I20240501 08:11:20.331543 11279 raft_consensus.cc:1067] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Signalling peer 6b0e60a3c4f74fdba1981d86fec4c529 to start an election
I20240501 08:11:20.332466 10174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "80636827cb8c427c8317f676d8790c70"
dest_uuid: "6b0e60a3c4f74fdba1981d86fec4c529"
from {username='slave'} at 127.6.56.66:33261
I20240501 08:11:20.332733 10174 raft_consensus.cc:482] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:20.332856 10174 raft_consensus.cc:3037] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:20.335630 10174 raft_consensus.cc:504] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:20.336360 10174 leader_election.cc:288] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [CANDIDATE]: Term 2 election: Requested vote from peers 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:20.336934 8535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80636827cb8c427c8317f676d8790c70" candidate_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
I20240501 08:11:20.337152 8535 raft_consensus.cc:3032] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:20.337255 8535 raft_consensus.cc:729] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:20.337525 8535 consensus_queue.cc:260] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:20.337849 8535 raft_consensus.cc:3037] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:20.340961 8535 raft_consensus.cc:2454] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b0e60a3c4f74fdba1981d86fec4c529 in term 2.
I20240501 08:11:20.341470 10109 leader_election.cc:302] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94, 6b0e60a3c4f74fdba1981d86fec4c529; no voters:
I20240501 08:11:20.341781 11159 raft_consensus.cc:2781] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:20.342301 11159 raft_consensus.cc:686] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 2 LEADER]: Becoming Leader. State: Replica: 6b0e60a3c4f74fdba1981d86fec4c529, State: Running, Role: LEADER
I20240501 08:11:20.342592 11159 consensus_queue.cc:237] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:20.345134 8260 catalog_manager.cc:5484] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 reported cstate change: term changed from 1 to 2, leader changed from 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) to 6b0e60a3c4f74fdba1981d86fec4c529 (127.6.56.71). New cstate: current_term: 2 leader_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:20.448318 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.454417 9764 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:20.462582 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.465231 8515 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:11:20.465862 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.489838 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.493773 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:20.586982 8535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "833a0a7dc6a047b19808085569e89368"
dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42276
I20240501 08:11:20.587193 8535 raft_consensus.cc:595] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:20.615695 10476 raft_consensus.cc:979] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: : Instructing follower 74bfd657aa5544d1a0925893d9121c7c to start an election
I20240501 08:11:20.615864 10476 raft_consensus.cc:1067] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Signalling peer 74bfd657aa5544d1a0925893d9121c7c to start an election
I20240501 08:11:20.616526 9784 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "833a0a7dc6a047b19808085569e89368"
dest_uuid: "74bfd657aa5544d1a0925893d9121c7c"
from {username='slave'} at 127.6.56.66:35811
I20240501 08:11:20.616744 9784 raft_consensus.cc:482] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:20.616854 9784 raft_consensus.cc:3037] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:20.619124 9784 raft_consensus.cc:504] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:20.619678 9784 leader_election.cc:288] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [CANDIDATE]: Term 2 election: Requested vote from peers 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:20.620249 8535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833a0a7dc6a047b19808085569e89368" candidate_uuid: "74bfd657aa5544d1a0925893d9121c7c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
I20240501 08:11:20.620460 8535 raft_consensus.cc:3032] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:20.620568 8535 raft_consensus.cc:729] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:20.620802 8535 consensus_queue.cc:260] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:20.621078 8535 raft_consensus.cc:3037] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:20.622897 8535 raft_consensus.cc:2454] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74bfd657aa5544d1a0925893d9121c7c in term 2.
I20240501 08:11:20.623322 9719 leader_election.cc:302] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94, 74bfd657aa5544d1a0925893d9121c7c; no voters:
I20240501 08:11:20.623661 11297 raft_consensus.cc:2781] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:20.624811 11297 raft_consensus.cc:686] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 2 LEADER]: Becoming Leader. State: Replica: 74bfd657aa5544d1a0925893d9121c7c, State: Running, Role: LEADER
I20240501 08:11:20.625139 11297 consensus_queue.cc:237] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:20.628468 8255 catalog_manager.cc:5484] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c reported cstate change: term changed from 1 to 2, leader changed from 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) to 74bfd657aa5544d1a0925893d9121c7c (127.6.56.68). New cstate: current_term: 2 leader_uuid: "74bfd657aa5544d1a0925893d9121c7c" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:20.687880 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.695164 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:20.706513 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.732682 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.742182 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.760844 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:20.765026 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:20.840847 8535 raft_consensus.cc:1261] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Refusing update from remote peer 6b0e60a3c4f74fdba1981d86fec4c529: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:20.841396 11159 consensus_queue.cc:1035] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:21.137892 8535 raft_consensus.cc:1261] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Refusing update from remote peer 74bfd657aa5544d1a0925893d9121c7c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:21.138543 11297 consensus_queue.cc:1035] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:21.151775 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.155267 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:21.163270 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.179973 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.190402 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.237341 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.259553 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:21.323093 10044 consensus_queue.cc:237] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:21.324380 11171 raft_consensus.cc:2932] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:21.329607 8259 catalog_manager.cc:5484] T d227b998061e46f59653167740f87fd0 P ac552fa18d104068846a832248e27bcc reported cstate change: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New cstate: current_term: 2 leader_uuid: "ac552fa18d104068846a832248e27bcc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:21.334873 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet d227b998061e46f59653167740f87fd0 with delete_type TABLET_DATA_TOMBSTONED (TS fdc83c70fcc04efda1a05f9d8199b9e9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36886
I20240501 08:11:21.335803 11430 tablet_replica.cc:330] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: stopping tablet replica
I20240501 08:11:21.336182 11430 raft_consensus.cc:2229] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:21.336479 11430 raft_consensus.cc:2258] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:21.337662 11430 ts_tablet_manager.cc:1875] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:21.342855 11430 ts_tablet_manager.cc:1888] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:21.343004 11430 log.cc:1196] T d227b998061e46f59653167740f87fd0 P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/d227b998061e46f59653167740f87fd0
I20240501 08:11:21.343964 8247 catalog_manager.cc:4830] TS fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073): tablet d227b998061e46f59653167740f87fd0 (table rebalance_test_table_2 [id=dee2da36183e48bf99429c14839719ed]) successfully deleted
I20240501 08:11:21.441433 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.452489 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.456113 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:21.458984 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.469588 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.472632 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:21.475765 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.602133 9784 consensus_queue.cc:237] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:21.603405 11399 raft_consensus.cc:2932] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:21.608021 8256 catalog_manager.cc:5484] T d71f0f9eeffd476ebd70487ff9a93d7c P 74bfd657aa5544d1a0925893d9121c7c reported cstate change: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New cstate: current_term: 2 leader_uuid: "74bfd657aa5544d1a0925893d9121c7c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:21.612067 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet d71f0f9eeffd476ebd70487ff9a93d7c with delete_type TABLET_DATA_TOMBSTONED (TS fdc83c70fcc04efda1a05f9d8199b9e9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36886
I20240501 08:11:21.612478 11430 tablet_replica.cc:330] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: stopping tablet replica
I20240501 08:11:21.612669 11430 raft_consensus.cc:2229] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:21.612850 11430 raft_consensus.cc:2258] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:21.613443 11430 ts_tablet_manager.cc:1875] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:21.618342 11430 ts_tablet_manager.cc:1888] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:21.618469 11430 log.cc:1196] T d71f0f9eeffd476ebd70487ff9a93d7c P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/d71f0f9eeffd476ebd70487ff9a93d7c
I20240501 08:11:21.619192 8247 catalog_manager.cc:4830] TS fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073): tablet d71f0f9eeffd476ebd70487ff9a93d7c (table rebalance_test_table_2 [id=dee2da36183e48bf99429c14839719ed]) successfully deleted
I20240501 08:11:21.725433 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.734439 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.737721 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.738670 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.742429 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.744892 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:21.762992 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:21.853642 10174 consensus_queue.cc:237] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:21.854806 11447 raft_consensus.cc:2932] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:21.860703 8257 catalog_manager.cc:5484] T e9cfe3dfff4848aabb0c7b420717130c P 6b0e60a3c4f74fdba1981d86fec4c529 reported cstate change: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New cstate: current_term: 2 leader_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:21.864435 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet e9cfe3dfff4848aabb0c7b420717130c with delete_type TABLET_DATA_TOMBSTONED (TS fdc83c70fcc04efda1a05f9d8199b9e9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36886
I20240501 08:11:21.864910 11430 tablet_replica.cc:330] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: stopping tablet replica
I20240501 08:11:21.865100 11430 raft_consensus.cc:2229] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:21.865314 11430 raft_consensus.cc:2258] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:21.866048 11430 ts_tablet_manager.cc:1875] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:21.871814 11430 ts_tablet_manager.cc:1888] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:21.871933 11430 log.cc:1196] T e9cfe3dfff4848aabb0c7b420717130c P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/e9cfe3dfff4848aabb0c7b420717130c
I20240501 08:11:21.872714 8247 catalog_manager.cc:4830] TS fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073): tablet e9cfe3dfff4848aabb0c7b420717130c (table rebalance_test_table_0 [id=7d90eb86e887478c9f151ae302c690e9]) successfully deleted
I20240501 08:11:21.959771 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.961855 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:21.976225 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.977188 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:21.991423 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.011699 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.019503 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:22.108028 9914 consensus_queue.cc:237] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:22.109390 11201 raft_consensus.cc:2932] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:22.115540 8256 catalog_manager.cc:5484] T e0ebe3885c6f4e3794bb61abf33dbb97 P e88366e322ed40e7bcff964b07cdeaf2 reported cstate change: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New cstate: current_term: 2 leader_uuid: "e88366e322ed40e7bcff964b07cdeaf2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:22.119654 8515 tablet_service.cc:1514] Processing DeleteTablet for tablet e0ebe3885c6f4e3794bb61abf33dbb97 with delete_type TABLET_DATA_TOMBSTONED (TS 2631692a8bb44abfb7d05022e6ae9b94 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55218
I20240501 08:11:22.120570 11588 tablet_replica.cc:330] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: stopping tablet replica
I20240501 08:11:22.121104 11588 raft_consensus.cc:2229] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:22.121481 11588 raft_consensus.cc:2258] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:22.123008 11588 ts_tablet_manager.cc:1875] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:22.129422 11588 ts_tablet_manager.cc:1888] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:22.129578 11588 log.cc:1196] T e0ebe3885c6f4e3794bb61abf33dbb97 P 2631692a8bb44abfb7d05022e6ae9b94: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0ebe3885c6f4e3794bb61abf33dbb97
I20240501 08:11:22.130520 8249 catalog_manager.cc:4830] TS 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791): tablet e0ebe3885c6f4e3794bb61abf33dbb97 (table rebalance_test_table_3 [id=9fbdcb332a0a4e90952a76a63564ab9c]) successfully deleted
I20240501 08:11:22.251510 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.253255 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:22.256898 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.265836 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.276602 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.284009 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:22.284510 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.382022 9644 consensus_queue.cc:237] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } }
I20240501 08:11:22.383750 11403 raft_consensus.cc:2932] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } } }
I20240501 08:11:22.389426 8256 catalog_manager.cc:5484] T 1f6207df29954aa3bfbc81304bd38827 P 0ac549325eca4b79be5c3861c96f23b2 reported cstate change: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New cstate: current_term: 2 leader_uuid: "0ac549325eca4b79be5c3861c96f23b2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0ac549325eca4b79be5c3861c96f23b2" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 42103 } } }
I20240501 08:11:22.393183 8515 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f6207df29954aa3bfbc81304bd38827 with delete_type TABLET_DATA_TOMBSTONED (TS 2631692a8bb44abfb7d05022e6ae9b94 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55218
I20240501 08:11:22.393568 11588 tablet_replica.cc:330] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: stopping tablet replica
I20240501 08:11:22.393755 11588 raft_consensus.cc:2229] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:22.393946 11588 raft_consensus.cc:2258] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:22.394829 11588 ts_tablet_manager.cc:1875] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:22.400614 11588 ts_tablet_manager.cc:1888] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:22.400805 11588 log.cc:1196] T 1f6207df29954aa3bfbc81304bd38827 P 2631692a8bb44abfb7d05022e6ae9b94: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/1f6207df29954aa3bfbc81304bd38827
I20240501 08:11:22.401687 8249 catalog_manager.cc:4830] TS 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791): tablet 1f6207df29954aa3bfbc81304bd38827 (table rebalance_test_table_4 [id=62a46f9934254f7a93b4a1ec0eef395b]) successfully deleted
I20240501 08:11:22.492389 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.506366 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.508332 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.513432 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:22.518779 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.531541 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.533361 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:22.649655 9914 consensus_queue.cc:237] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } }
I20240501 08:11:22.651129 11697 raft_consensus.cc:2932] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:22.655632 8257 catalog_manager.cc:5484] T 532f21629fae48408e569bdcd04c2cbd P e88366e322ed40e7bcff964b07cdeaf2 reported cstate change: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New cstate: current_term: 2 leader_uuid: "e88366e322ed40e7bcff964b07cdeaf2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e88366e322ed40e7bcff964b07cdeaf2" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 33185 } } }
I20240501 08:11:22.659466 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet 532f21629fae48408e569bdcd04c2cbd with delete_type TABLET_DATA_TOMBSTONED (TS fdc83c70fcc04efda1a05f9d8199b9e9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36886
I20240501 08:11:22.660183 11701 tablet_replica.cc:330] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: stopping tablet replica
I20240501 08:11:22.660426 11701 raft_consensus.cc:2229] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:22.660669 11701 raft_consensus.cc:2258] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:22.661383 11701 ts_tablet_manager.cc:1875] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:22.667290 11701 ts_tablet_manager.cc:1888] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:22.667418 11701 log.cc:1196] T 532f21629fae48408e569bdcd04c2cbd P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/532f21629fae48408e569bdcd04c2cbd
I20240501 08:11:22.668220 8247 catalog_manager.cc:4830] TS fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073): tablet 532f21629fae48408e569bdcd04c2cbd (table rebalance_test_table_0 [id=7d90eb86e887478c9f151ae302c690e9]) successfully deleted
I20240501 08:11:22.776506 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.776562 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:22.777743 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.780354 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.786312 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:22.797173 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.807813 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:22.896773 10044 consensus_queue.cc:237] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } }
I20240501 08:11:22.898592 11704 raft_consensus.cc:2932] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:22.903129 8256 catalog_manager.cc:5484] T 493df0048a8740f190d7d126e70e8bc5 P ac552fa18d104068846a832248e27bcc reported cstate change: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New cstate: current_term: 2 leader_uuid: "ac552fa18d104068846a832248e27bcc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ac552fa18d104068846a832248e27bcc" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 33819 } } }
I20240501 08:11:22.906790 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet 493df0048a8740f190d7d126e70e8bc5 with delete_type TABLET_DATA_TOMBSTONED (TS fdc83c70fcc04efda1a05f9d8199b9e9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36886
I20240501 08:11:22.907155 11701 tablet_replica.cc:330] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: stopping tablet replica
I20240501 08:11:22.907310 11701 raft_consensus.cc:2229] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:22.907493 11701 raft_consensus.cc:2258] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:22.908113 11701 ts_tablet_manager.cc:1875] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:22.913237 11701 ts_tablet_manager.cc:1888] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:22.913389 11701 log.cc:1196] T 493df0048a8740f190d7d126e70e8bc5 P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/493df0048a8740f190d7d126e70e8bc5
I20240501 08:11:22.914177 8247 catalog_manager.cc:4830] TS fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073): tablet 493df0048a8740f190d7d126e70e8bc5 (table rebalance_test_table_4 [id=62a46f9934254f7a93b4a1ec0eef395b]) successfully deleted
I20240501 08:11:23.000943 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.010125 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.020784 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:23.039525 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.057333 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.062924 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.094925 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:23.195006 10174 consensus_queue.cc:237] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } }
I20240501 08:11:23.196547 11447 raft_consensus.cc:2932] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:23.214028 8261 catalog_manager.cc:5484] T 80636827cb8c427c8317f676d8790c70 P 6b0e60a3c4f74fdba1981d86fec4c529 reported cstate change: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New cstate: current_term: 2 leader_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6b0e60a3c4f74fdba1981d86fec4c529" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 42449 } } }
I20240501 08:11:23.218868 8515 tablet_service.cc:1514] Processing DeleteTablet for tablet 80636827cb8c427c8317f676d8790c70 with delete_type TABLET_DATA_TOMBSTONED (TS 2631692a8bb44abfb7d05022e6ae9b94 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55218
I20240501 08:11:23.219913 11809 tablet_replica.cc:330] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: stopping tablet replica
I20240501 08:11:23.220170 11809 raft_consensus.cc:2229] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:23.220536 11809 raft_consensus.cc:2258] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:23.221539 11809 ts_tablet_manager.cc:1875] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:23.230064 11809 ts_tablet_manager.cc:1888] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:23.230772 11809 log.cc:1196] T 80636827cb8c427c8317f676d8790c70 P 2631692a8bb44abfb7d05022e6ae9b94: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/80636827cb8c427c8317f676d8790c70
I20240501 08:11:23.231925 8249 catalog_manager.cc:4830] TS 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791): tablet 80636827cb8c427c8317f676d8790c70 (table rebalance_test_table_3 [id=9fbdcb332a0a4e90952a76a63564ab9c]) successfully deleted
I20240501 08:11:23.365514 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.366986 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:23.391901 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.392195 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.406126 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.428491 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.465121 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:23.536839 9784 consensus_queue.cc:237] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } }
I20240501 08:11:23.538141 11791 raft_consensus.cc:2932] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:23.546765 8261 catalog_manager.cc:5484] T 833a0a7dc6a047b19808085569e89368 P 74bfd657aa5544d1a0925893d9121c7c reported cstate change: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New cstate: current_term: 2 leader_uuid: "74bfd657aa5544d1a0925893d9121c7c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "74bfd657aa5544d1a0925893d9121c7c" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 38709 } } }
I20240501 08:11:23.551497 8515 tablet_service.cc:1514] Processing DeleteTablet for tablet 833a0a7dc6a047b19808085569e89368 with delete_type TABLET_DATA_TOMBSTONED (TS 2631692a8bb44abfb7d05022e6ae9b94 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55218
I20240501 08:11:23.551903 11809 tablet_replica.cc:330] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: stopping tablet replica
I20240501 08:11:23.552130 11809 raft_consensus.cc:2229] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:23.552366 11809 raft_consensus.cc:2258] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:23.553124 11809 ts_tablet_manager.cc:1875] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:23.560074 11809 ts_tablet_manager.cc:1888] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:23.560350 11809 log.cc:1196] T 833a0a7dc6a047b19808085569e89368 P 2631692a8bb44abfb7d05022e6ae9b94: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/833a0a7dc6a047b19808085569e89368
I20240501 08:11:23.561452 8249 catalog_manager.cc:4830] TS 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791): tablet 833a0a7dc6a047b19808085569e89368 (table rebalance_test_table_1 [id=a622f24ae9504f47acdb022fb9714306]) successfully deleted
I20240501 08:11:23.920791 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.926142 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.952795 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:23.954203 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.968101 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:23.977151 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:24.006106 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:24.165699 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:24.169291 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:24.177901 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:24.183971 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:24.188596 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:24.191821 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:24.201709 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:24.954200 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:24.975780 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:24.976878 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:24.981424 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.016741 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:25.046741 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.054467 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.215729 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.223697 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:25.234920 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.237556 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.249521 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.250872 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.294976 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:25.559466 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:25.559612 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.575146 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.596473 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.599269 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.609905 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:25.610886 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
0276005c13714ca09390ff319a3ff96a | 127.6.56.126:36943 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.56.84:40685 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
0ac549325eca4b79be5c3861c96f23b2 | 127.6.56.67:42103 | HEALTHY | <none> | 2 | 0
2631692a8bb44abfb7d05022e6ae9b94 | 127.6.56.66:40791 | HEALTHY | <none> | 3 | 0
6b0e60a3c4f74fdba1981d86fec4c529 | 127.6.56.71:42449 | HEALTHY | <none> | 2 | 0
74bfd657aa5544d1a0925893d9121c7c | 127.6.56.68:38709 | HEALTHY | <none> | 2 | 0
ac552fa18d104068846a832248e27bcc | 127.6.56.70:33819 | HEALTHY | <none> | 2 | 0
e88366e322ed40e7bcff964b07cdeaf2 | 127.6.56.69:33185 | HEALTHY | <none> | 2 | 0
fdc83c70fcc04efda1a05f9d8199b9e9 | 127.6.56.65:46073 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.6.56.65 | experimental | 127.6.56.65:46073
local_ip_for_outbound_sockets | 127.6.56.66 | experimental | 127.6.56.66:40791
local_ip_for_outbound_sockets | 127.6.56.67 | experimental | 127.6.56.67:42103
local_ip_for_outbound_sockets | 127.6.56.68 | experimental | 127.6.56.68:38709
local_ip_for_outbound_sockets | 127.6.56.69 | experimental | 127.6.56.69:33185
local_ip_for_outbound_sockets | 127.6.56.70 | experimental | 127.6.56.70:33819
local_ip_for_outbound_sockets | 127.6.56.71 | experimental | 127.6.56.71:42449
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.56.65:46073
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.56.66:40791
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.56.67:42103
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.56.68:38709
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.56.69:33185
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.56.70:33819
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.56.71:42449
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.56.84:40685 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 08:11:25.686360 6369 log_verifier.cc:126] Checking tablet 197210fe0ae04e28be414b5a08d15b7c
I20240501 08:11:25.696720 6369 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 197210fe0ae04e28be414b5a08d15b7c
I20240501 08:11:25.696861 6369 log_verifier.cc:126] Checking tablet 1f6207df29954aa3bfbc81304bd38827
I20240501 08:11:25.709928 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1f6207df29954aa3bfbc81304bd38827
I20240501 08:11:25.710041 6369 log_verifier.cc:126] Checking tablet 33975146f8f5488ab1818546e7cadca0
I20240501 08:11:25.722299 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 33975146f8f5488ab1818546e7cadca0
I20240501 08:11:25.722407 6369 log_verifier.cc:126] Checking tablet 3cc149e11b784b929ce06059690b4a7e
I20240501 08:11:25.734232 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3cc149e11b784b929ce06059690b4a7e
I20240501 08:11:25.734337 6369 log_verifier.cc:126] Checking tablet 3f4f5ffc5f44469d9e65db69546d836f
I20240501 08:11:25.745934 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3f4f5ffc5f44469d9e65db69546d836f
I20240501 08:11:25.746035 6369 log_verifier.cc:126] Checking tablet 493df0048a8740f190d7d126e70e8bc5
I20240501 08:11:25.758172 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 493df0048a8740f190d7d126e70e8bc5
I20240501 08:11:25.758276 6369 log_verifier.cc:126] Checking tablet 532f21629fae48408e569bdcd04c2cbd
I20240501 08:11:25.770321 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 532f21629fae48408e569bdcd04c2cbd
I20240501 08:11:25.770447 6369 log_verifier.cc:126] Checking tablet 80636827cb8c427c8317f676d8790c70
I20240501 08:11:25.782202 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 80636827cb8c427c8317f676d8790c70
I20240501 08:11:25.782321 6369 log_verifier.cc:126] Checking tablet 833a0a7dc6a047b19808085569e89368
I20240501 08:11:25.794746 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 833a0a7dc6a047b19808085569e89368
I20240501 08:11:25.794864 6369 log_verifier.cc:126] Checking tablet 9e92c6c0f9114282b832ecaddffea60a
I20240501 08:11:25.806767 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9e92c6c0f9114282b832ecaddffea60a
I20240501 08:11:25.806879 6369 log_verifier.cc:126] Checking tablet d227b998061e46f59653167740f87fd0
I20240501 08:11:25.819525 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d227b998061e46f59653167740f87fd0
I20240501 08:11:25.819640 6369 log_verifier.cc:126] Checking tablet d71f0f9eeffd476ebd70487ff9a93d7c
I20240501 08:11:25.832518 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d71f0f9eeffd476ebd70487ff9a93d7c
I20240501 08:11:25.832636 6369 log_verifier.cc:126] Checking tablet e0ebe3885c6f4e3794bb61abf33dbb97
I20240501 08:11:25.844779 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e0ebe3885c6f4e3794bb61abf33dbb97
I20240501 08:11:25.844895 6369 log_verifier.cc:126] Checking tablet e836061c829848caa03078113b94c44e
I20240501 08:11:25.856684 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e836061c829848caa03078113b94c44e
I20240501 08:11:25.856796 6369 log_verifier.cc:126] Checking tablet e9cfe3dfff4848aabb0c7b420717130c
I20240501 08:11:25.868800 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e9cfe3dfff4848aabb0c7b420717130c
I20240501 08:11:25.868934 6369 log_verifier.cc:126] Checking tablet fe8bf89936b04faca8c7ab15ce402e8f
I20240501 08:11:25.880867 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fe8bf89936b04faca8c7ab15ce402e8f
I20240501 08:11:25.883865 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.72:0
--local_ip_for_outbound_sockets=127.6.56.72
--webserver_interface=127.6.56.72
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--builtin_ntp_servers=127.6.56.84:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:11:26.235626 12083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:26.235916 12083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:26.236008 12083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:26.243628 12083 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:26.243769 12083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:26.243885 12083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.72
I20240501 08:11:26.253412 12083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.56.72
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36943
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.72
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:26.254657 12083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:26.256332 12083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:26.269472 12089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:26.269949 12088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:26.270869 12091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:26.271742 12083 server_base.cc:1034] running on GCE node
I20240501 08:11:26.272401 12083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:26.273320 12083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:26.274505 12083 hybrid_clock.cc:648] HybridClock initialized: now 1714551086274450 us; error 50 us; skew 500 ppm
I20240501 08:11:26.274814 12083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:26.277166 12083 webserver.cc:460] Webserver started at http://127.6.56.72:42581/ using document root <none> and password file <none>
I20240501 08:11:26.277899 12083 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:26.278015 12083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:26.278338 12083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:26.280944 12083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "8438aa754cbb451db2c2b7e8211699ec"
format_stamp: "Formatted at 2024-05-01 08:11:26 on dist-test-slave-f1nk"
I20240501 08:11:26.281721 12083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "8438aa754cbb451db2c2b7e8211699ec"
format_stamp: "Formatted at 2024-05-01 08:11:26 on dist-test-slave-f1nk"
I20240501 08:11:26.286818 12083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:26.290604 12097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.291927 12083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:26.292091 12083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8438aa754cbb451db2c2b7e8211699ec"
format_stamp: "Formatted at 2024-05-01 08:11:26 on dist-test-slave-f1nk"
I20240501 08:11:26.292272 12083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:26.317129 12083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:26.318015 12083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:26.318348 12083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:26.319286 12083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:26.320751 12083 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:26.320838 12083 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.320921 12083 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:26.320974 12083 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:26.356832 12083 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.72:33821
I20240501 08:11:26.356880 12209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.72:33821 every 8 connection(s)
I20240501 08:11:26.358119 12083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:11:26.364084 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 12083
I20240501 08:11:26.364360 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:11:26.373169 12210 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36943
I20240501 08:11:26.373499 12210 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:26.374271 12210 heartbeater.cc:511] Master 127.6.56.126:36943 requested a full tablet report, sending...
I20240501 08:11:26.375705 8259 ts_manager.cc:194] Registered new tserver with Master: 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72:33821)
I20240501 08:11:26.376539 8259 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.72:48835
I20240501 08:11:26.860667 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:26.879244 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:26.882042 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:26.903071 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:26.904492 12145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:26.924566 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:26.945336 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:26.969204 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:27.148962 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:27.149442 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:27.179884 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:27.180353 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:27.180969 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:27.202749 12145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:27.224321 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:27.237483 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:27.328047 8403 consensus_queue.cc:237] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
W20240501 08:11:27.337165 8340 consensus_peers.cc:485] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 -> Peer 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72:33821): Couldn't send request to peer 8438aa754cbb451db2c2b7e8211699ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe8bf89936b04faca8c7ab15ce402e8f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:27.379528 12210 heartbeater.cc:503] Master 127.6.56.126:36943 was elected leader, sending a full tablet report...
I20240501 08:11:27.911747 12316 ts_tablet_manager.cc:902] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: Initiating tablet copy from peer fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:27.913435 12316 tablet_copy_client.cc:308] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46073
I20240501 08:11:27.920922 8424 tablet_copy_service.cc:140] P fdc83c70fcc04efda1a05f9d8199b9e9: Received BeginTabletCopySession request for tablet fe8bf89936b04faca8c7ab15ce402e8f from peer 8438aa754cbb451db2c2b7e8211699ec ({username='slave'} at 127.6.56.72:38111)
I20240501 08:11:27.921113 8424 tablet_copy_service.cc:161] P fdc83c70fcc04efda1a05f9d8199b9e9: Beginning new tablet copy session on tablet fe8bf89936b04faca8c7ab15ce402e8f from peer 8438aa754cbb451db2c2b7e8211699ec at {username='slave'} at 127.6.56.72:38111: session id = 8438aa754cbb451db2c2b7e8211699ec-fe8bf89936b04faca8c7ab15ce402e8f
I20240501 08:11:27.923166 8424 tablet_copy_source_session.cc:205] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:27.927479 12316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8bf89936b04faca8c7ab15ce402e8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:27.937685 12316 tablet_copy_client.cc:783] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:27.938015 12316 tablet_copy_client.cc:647] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:27.940150 12316 tablet_copy_client.cc:523] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:27.943367 12316 tablet_bootstrap.cc:492] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: Bootstrap starting.
I20240501 08:11:27.953513 12316 log.cc:824] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:27.986811 12316 tablet_bootstrap.cc:492] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:27.987440 12316 tablet_bootstrap.cc:492] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: Bootstrap complete.
I20240501 08:11:27.987892 12316 ts_tablet_manager.cc:1367] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: Time spent bootstrapping tablet: real 0.045s user 0.040s sys 0.004s
I20240501 08:11:27.992453 12316 raft_consensus.cc:348] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } }
I20240501 08:11:27.993283 12316 raft_consensus.cc:729] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8438aa754cbb451db2c2b7e8211699ec, State: Initialized, Role: FOLLOWER
I20240501 08:11:27.993711 12316 consensus_queue.cc:260] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:27.995280 12316 ts_tablet_manager.cc:1398] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec: Time spent starting tablet: real 0.007s user 0.010s sys 0.000s
I20240501 08:11:27.996443 8424 tablet_copy_service.cc:342] P fdc83c70fcc04efda1a05f9d8199b9e9: Request end of tablet copy session 8438aa754cbb451db2c2b7e8211699ec-fe8bf89936b04faca8c7ab15ce402e8f received from {username='slave'} at 127.6.56.72:38111
I20240501 08:11:27.996590 8424 tablet_copy_service.cc:433] P fdc83c70fcc04efda1a05f9d8199b9e9: ending tablet copy session 8438aa754cbb451db2c2b7e8211699ec-fe8bf89936b04faca8c7ab15ce402e8f on tablet fe8bf89936b04faca8c7ab15ce402e8f with peer 8438aa754cbb451db2c2b7e8211699ec
I20240501 08:11:28.300552 12165 raft_consensus.cc:1203] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:28.302388 12325 raft_consensus.cc:2932] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:28.303306 12165 raft_consensus.cc:2932] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:28.308444 8260 catalog_manager.cc:5484] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 reported cstate change: config changed from index -1 to 4, VOTER 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:28.337610 8535 consensus_queue.cc:237] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
W20240501 08:11:28.345953 8471 consensus_peers.cc:485] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 -> Peer 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72:33821): Couldn't send request to peer 8438aa754cbb451db2c2b7e8211699ec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f4f5ffc5f44469d9e65db69546d836f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:11:28.852342 12354 ts_tablet_manager.cc:902] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: Initiating tablet copy from peer 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:28.853405 12354 tablet_copy_client.cc:308] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:40791
I20240501 08:11:28.861506 8555 tablet_copy_service.cc:140] P 2631692a8bb44abfb7d05022e6ae9b94: Received BeginTabletCopySession request for tablet 3f4f5ffc5f44469d9e65db69546d836f from peer 8438aa754cbb451db2c2b7e8211699ec ({username='slave'} at 127.6.56.72:40713)
I20240501 08:11:28.861730 8555 tablet_copy_service.cc:161] P 2631692a8bb44abfb7d05022e6ae9b94: Beginning new tablet copy session on tablet 3f4f5ffc5f44469d9e65db69546d836f from peer 8438aa754cbb451db2c2b7e8211699ec at {username='slave'} at 127.6.56.72:40713: session id = 8438aa754cbb451db2c2b7e8211699ec-3f4f5ffc5f44469d9e65db69546d836f
I20240501 08:11:28.865373 8555 tablet_copy_source_session.cc:205] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:11:28.867053 12354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f4f5ffc5f44469d9e65db69546d836f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:28.871894 12354 tablet_copy_client.cc:783] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: tablet copy: Starting download of 0 data blocks...
I20240501 08:11:28.872128 12354 tablet_copy_client.cc:647] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: tablet copy: Starting download of 1 WAL segments...
I20240501 08:11:28.873853 12354 tablet_copy_client.cc:523] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:11:28.876236 12354 tablet_bootstrap.cc:492] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: Bootstrap starting.
I20240501 08:11:28.905325 12354 tablet_bootstrap.cc:492] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 08:11:28.905745 12354 tablet_bootstrap.cc:492] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: Bootstrap complete.
I20240501 08:11:28.905997 12354 ts_tablet_manager.cc:1367] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: Time spent bootstrapping tablet: real 0.030s user 0.026s sys 0.004s
I20240501 08:11:28.907258 12354 raft_consensus.cc:348] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } }
I20240501 08:11:28.907625 12354 raft_consensus.cc:729] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8438aa754cbb451db2c2b7e8211699ec, State: Initialized, Role: FOLLOWER
I20240501 08:11:28.907811 12354 consensus_queue.cc:260] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:28.908572 12354 ts_tablet_manager.cc:1398] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240501 08:11:28.909456 8555 tablet_copy_service.cc:342] P 2631692a8bb44abfb7d05022e6ae9b94: Request end of tablet copy session 8438aa754cbb451db2c2b7e8211699ec-3f4f5ffc5f44469d9e65db69546d836f received from {username='slave'} at 127.6.56.72:40713
I20240501 08:11:28.909619 8555 tablet_copy_service.cc:433] P 2631692a8bb44abfb7d05022e6ae9b94: ending tablet copy session 8438aa754cbb451db2c2b7e8211699ec-3f4f5ffc5f44469d9e65db69546d836f on tablet 3f4f5ffc5f44469d9e65db69546d836f with peer 8438aa754cbb451db2c2b7e8211699ec
I20240501 08:11:29.381510 12165 raft_consensus.cc:1203] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:11:29.382844 12365 raft_consensus.cc:2932] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:29.383493 12165 raft_consensus.cc:2932] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:29.386636 12365 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:36840 (request call id 0) took 1049 ms. Trace:
I20240501 08:11:29.386878 12365 rpcz_store.cc:276] 0501 08:11:28.337062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0501 08:11:28.337134 (+ 72us) service_pool.cc:225] Handling call
0501 08:11:28.338806 (+ 1672us) log.cc:842] Serialized 254 byte log entry
0501 08:11:29.386583 (+1047777us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":74,"raft.queue_time_us":389,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":243,"thread_start_us":510,"threads_started":2,"wal-append.queue_time_us":328,"wal-append.run_cpu_time_us":587,"wal-append.run_wall_time_us":1000606}
I20240501 08:11:29.389585 8256 catalog_manager.cc:5484] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec reported cstate change: config changed from index -1 to 4, VOTER 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "2631692a8bb44abfb7d05022e6ae9b94" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:29.499861 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.499861 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:29.510859 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.525019 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.538005 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:29.539871 12145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:29.548408 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.556241 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.661239 8535 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f4f5ffc5f44469d9e65db69546d836f"
dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36868
I20240501 08:11:29.661458 8535 raft_consensus.cc:595] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:29.749174 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.765362 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.765554 8515 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:29.780181 12145 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:11:29.789916 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.792693 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.796201 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.823559 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:29.899708 12469 raft_consensus.cc:979] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: : Instructing follower 8438aa754cbb451db2c2b7e8211699ec to start an election
I20240501 08:11:29.899890 12469 raft_consensus.cc:1067] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Signalling peer 8438aa754cbb451db2c2b7e8211699ec to start an election
I20240501 08:11:29.900780 12165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f4f5ffc5f44469d9e65db69546d836f"
dest_uuid: "8438aa754cbb451db2c2b7e8211699ec"
from {username='slave'} at 127.6.56.66:49325
I20240501 08:11:29.901597 12165 raft_consensus.cc:482] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:29.901798 12165 raft_consensus.cc:3037] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:29.903985 12165 raft_consensus.cc:504] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:29.905380 12165 leader_election.cc:288] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [CANDIDATE]: Term 2 election: Requested vote from peers 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791)
I20240501 08:11:29.905884 8535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4f5ffc5f44469d9e65db69546d836f" candidate_uuid: "8438aa754cbb451db2c2b7e8211699ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "2631692a8bb44abfb7d05022e6ae9b94"
I20240501 08:11:29.906090 8535 raft_consensus.cc:3032] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:29.906167 8535 raft_consensus.cc:729] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2631692a8bb44abfb7d05022e6ae9b94, State: Running, Role: LEADER
I20240501 08:11:29.906455 8535 consensus_queue.cc:260] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:29.906705 8535 raft_consensus.cc:3037] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:29.908681 8535 raft_consensus.cc:2454] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8438aa754cbb451db2c2b7e8211699ec in term 2.
I20240501 08:11:29.909219 12100 leader_election.cc:302] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2631692a8bb44abfb7d05022e6ae9b94, 8438aa754cbb451db2c2b7e8211699ec; no voters:
I20240501 08:11:29.909862 12470 raft_consensus.cc:2781] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:29.910323 12470 raft_consensus.cc:686] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 2 LEADER]: Becoming Leader. State: Replica: 8438aa754cbb451db2c2b7e8211699ec, State: Running, Role: LEADER
I20240501 08:11:29.910900 12470 consensus_queue.cc:237] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:29.914678 8260 catalog_manager.cc:5484] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec reported cstate change: term changed from 1 to 2, leader changed from 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) to 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72). New cstate: current_term: 2 leader_uuid: "8438aa754cbb451db2c2b7e8211699ec" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:29.994138 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.997848 8515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:29.999580 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.014889 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.017103 12145 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:30.024279 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.026975 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.045336 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:30.123899 8403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe8bf89936b04faca8c7ab15ce402e8f"
dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33078
I20240501 08:11:30.124102 8403 raft_consensus.cc:595] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:11:30.214891 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.227479 8515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.231473 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.251000 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.251667 12145 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:11:30.251737 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.261952 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.274950 8383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:11:30.333341 12541 raft_consensus.cc:979] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: : Instructing follower 8438aa754cbb451db2c2b7e8211699ec to start an election
I20240501 08:11:30.333508 12541 raft_consensus.cc:1067] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Signalling peer 8438aa754cbb451db2c2b7e8211699ec to start an election
I20240501 08:11:30.334391 12165 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe8bf89936b04faca8c7ab15ce402e8f"
dest_uuid: "8438aa754cbb451db2c2b7e8211699ec"
from {username='slave'} at 127.6.56.65:42993
I20240501 08:11:30.334662 12165 raft_consensus.cc:482] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:11:30.334786 12165 raft_consensus.cc:3037] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:30.337989 12165 raft_consensus.cc:504] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:30.338848 12165 leader_election.cc:288] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [CANDIDATE]: Term 2 election: Requested vote from peers fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073)
I20240501 08:11:30.339455 8403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe8bf89936b04faca8c7ab15ce402e8f" candidate_uuid: "8438aa754cbb451db2c2b7e8211699ec" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9"
I20240501 08:11:30.339674 8403 raft_consensus.cc:3032] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:11:30.339771 8403 raft_consensus.cc:729] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fdc83c70fcc04efda1a05f9d8199b9e9, State: Running, Role: LEADER
I20240501 08:11:30.340027 8403 consensus_queue.cc:260] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:30.340369 8403 raft_consensus.cc:3037] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:11:30.342991 8403 raft_consensus.cc:2454] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8438aa754cbb451db2c2b7e8211699ec in term 2.
I20240501 08:11:30.343492 12098 leader_election.cc:302] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 8438aa754cbb451db2c2b7e8211699ec, fdc83c70fcc04efda1a05f9d8199b9e9; no voters:
I20240501 08:11:30.343816 12470 raft_consensus.cc:2781] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:11:30.343984 12470 raft_consensus.cc:686] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 2 LEADER]: Becoming Leader. State: Replica: 8438aa754cbb451db2c2b7e8211699ec, State: Running, Role: LEADER
I20240501 08:11:30.344260 12470 consensus_queue.cc:237] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:30.346918 8257 catalog_manager.cc:5484] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec reported cstate change: term changed from 1 to 2, leader changed from fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) to 8438aa754cbb451db2c2b7e8211699ec (127.6.56.72). New cstate: current_term: 2 leader_uuid: "8438aa754cbb451db2c2b7e8211699ec" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 } } peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:30.362354 8535 raft_consensus.cc:1261] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Refusing update from remote peer 8438aa754cbb451db2c2b7e8211699ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:30.363204 12470 consensus_queue.cc:1035] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "2631692a8bb44abfb7d05022e6ae9b94" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:30.707418 8515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.708148 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.724876 12145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.725036 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.725927 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.740195 8403 raft_consensus.cc:1261] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Refusing update from remote peer 8438aa754cbb451db2c2b7e8211699ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:11:30.740917 12470 consensus_queue.cc:1035] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc83c70fcc04efda1a05f9d8199b9e9" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:11:30.785954 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.796020 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.819909 8383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.883455 12165 consensus_queue.cc:237] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:30.884223 12471 raft_consensus.cc:2932] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:30.888659 8256 catalog_manager.cc:5484] T 3f4f5ffc5f44469d9e65db69546d836f P 8438aa754cbb451db2c2b7e8211699ec reported cstate change: config changed from index 4 to 6, VOTER 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66) evicted. New cstate: current_term: 2 leader_uuid: "8438aa754cbb451db2c2b7e8211699ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:30.893605 8515 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f4f5ffc5f44469d9e65db69546d836f with delete_type TABLET_DATA_TOMBSTONED (TS 2631692a8bb44abfb7d05022e6ae9b94 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:55218
I20240501 08:11:30.894207 12620 tablet_replica.cc:330] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: stopping tablet replica
I20240501 08:11:30.894412 12620 raft_consensus.cc:2229] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:30.894616 12620 raft_consensus.cc:2258] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:30.896345 12620 ts_tablet_manager.cc:1875] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:30.902658 12620 ts_tablet_manager.cc:1888] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:30.902767 12620 log.cc:1196] T 3f4f5ffc5f44469d9e65db69546d836f P 2631692a8bb44abfb7d05022e6ae9b94: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/3f4f5ffc5f44469d9e65db69546d836f
I20240501 08:11:30.903517 8249 catalog_manager.cc:4830] TS 2631692a8bb44abfb7d05022e6ae9b94 (127.6.56.66:40791): tablet 3f4f5ffc5f44469d9e65db69546d836f (table rebalance_test_table_2 [id=dee2da36183e48bf99429c14839719ed]) successfully deleted
I20240501 08:11:30.991617 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:30.998701 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.006726 8383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.011250 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.015781 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.024761 12145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.041849 8515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.044306 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.142270 12165 consensus_queue.cc:237] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } }
I20240501 08:11:31.143092 12471 raft_consensus.cc:2932] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:31.147063 8257 catalog_manager.cc:5484] T fe8bf89936b04faca8c7ab15ce402e8f P 8438aa754cbb451db2c2b7e8211699ec reported cstate change: config changed from index 4 to 6, VOTER fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65) evicted. New cstate: current_term: 2 leader_uuid: "8438aa754cbb451db2c2b7e8211699ec" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "8438aa754cbb451db2c2b7e8211699ec" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 33821 } } }
I20240501 08:11:31.150481 8383 tablet_service.cc:1514] Processing DeleteTablet for tablet fe8bf89936b04faca8c7ab15ce402e8f with delete_type TABLET_DATA_TOMBSTONED (TS fdc83c70fcc04efda1a05f9d8199b9e9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36886
I20240501 08:11:31.151087 12669 tablet_replica.cc:330] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: stopping tablet replica
I20240501 08:11:31.151259 12669 raft_consensus.cc:2229] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:11:31.151437 12669 raft_consensus.cc:2258] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:11:31.152906 12669 ts_tablet_manager.cc:1875] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:11:31.157315 12669 ts_tablet_manager.cc:1888] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:11:31.157429 12669 log.cc:1196] T fe8bf89936b04faca8c7ab15ce402e8f P fdc83c70fcc04efda1a05f9d8199b9e9: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe8bf89936b04faca8c7ab15ce402e8f
I20240501 08:11:31.158170 8247 catalog_manager.cc:4830] TS fdc83c70fcc04efda1a05f9d8199b9e9 (127.6.56.65:46073): tablet fe8bf89936b04faca8c7ab15ce402e8f (table rebalance_test_table_4 [id=62a46f9934254f7a93b4a1ec0eef395b]) successfully deleted
I20240501 08:11:31.475601 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.490015 8515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.490809 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.496428 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.505456 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.532582 12145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.551162 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.591374 8383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.755954 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.764720 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.768846 12145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.790169 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.794314 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.799368 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.800882 8515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:31.834319 8383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:32.117481 9623 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:32.120558 8515 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:32.125249 10154 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:32.152024 10024 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:32.157423 9764 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:32.158608 12145 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:32.170459 9894 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:11:32.192155 8383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
0276005c13714ca09390ff319a3ff96a | 127.6.56.126:36943 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.56.84:40685 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
0ac549325eca4b79be5c3861c96f23b2 | 127.6.56.67:42103 | HEALTHY | <none> | 2 | 0
2631692a8bb44abfb7d05022e6ae9b94 | 127.6.56.66:40791 | HEALTHY | <none> | 2 | 0
6b0e60a3c4f74fdba1981d86fec4c529 | 127.6.56.71:42449 | HEALTHY | <none> | 2 | 0
74bfd657aa5544d1a0925893d9121c7c | 127.6.56.68:38709 | HEALTHY | <none> | 2 | 0
8438aa754cbb451db2c2b7e8211699ec | 127.6.56.72:33821 | HEALTHY | <none> | 2 | 0
ac552fa18d104068846a832248e27bcc | 127.6.56.70:33819 | HEALTHY | <none> | 2 | 0
e88366e322ed40e7bcff964b07cdeaf2 | 127.6.56.69:33185 | HEALTHY | <none> | 2 | 0
fdc83c70fcc04efda1a05f9d8199b9e9 | 127.6.56.65:46073 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.6.56.65 | experimental | 127.6.56.65:46073
local_ip_for_outbound_sockets | 127.6.56.66 | experimental | 127.6.56.66:40791
local_ip_for_outbound_sockets | 127.6.56.67 | experimental | 127.6.56.67:42103
local_ip_for_outbound_sockets | 127.6.56.68 | experimental | 127.6.56.68:38709
local_ip_for_outbound_sockets | 127.6.56.69 | experimental | 127.6.56.69:33185
local_ip_for_outbound_sockets | 127.6.56.70 | experimental | 127.6.56.70:33819
local_ip_for_outbound_sockets | 127.6.56.71 | experimental | 127.6.56.71:42449
local_ip_for_outbound_sockets | 127.6.56.72 | experimental | 127.6.56.72:33821
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.56.65:46073
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.56.66:40791
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.56.67:42103
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.56.68:38709
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.56.69:33185
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.56.70:33819
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.56.71:42449
server_dump_info_path | /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalanceParamTest.Rebalance_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.6.56.72:33821
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.6.56.84:40685 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 08:11:32.257695 6369 log_verifier.cc:126] Checking tablet 197210fe0ae04e28be414b5a08d15b7c
I20240501 08:11:32.263065 6369 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 197210fe0ae04e28be414b5a08d15b7c
I20240501 08:11:32.263154 6369 log_verifier.cc:126] Checking tablet 1f6207df29954aa3bfbc81304bd38827
I20240501 08:11:32.273360 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 1f6207df29954aa3bfbc81304bd38827
I20240501 08:11:32.273468 6369 log_verifier.cc:126] Checking tablet 33975146f8f5488ab1818546e7cadca0
I20240501 08:11:32.282729 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 33975146f8f5488ab1818546e7cadca0
I20240501 08:11:32.282817 6369 log_verifier.cc:126] Checking tablet 3cc149e11b784b929ce06059690b4a7e
I20240501 08:11:32.291980 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3cc149e11b784b929ce06059690b4a7e
I20240501 08:11:32.292078 6369 log_verifier.cc:126] Checking tablet 3f4f5ffc5f44469d9e65db69546d836f
I20240501 08:11:32.307520 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3f4f5ffc5f44469d9e65db69546d836f
I20240501 08:11:32.307621 6369 log_verifier.cc:126] Checking tablet 493df0048a8740f190d7d126e70e8bc5
I20240501 08:11:32.317324 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 493df0048a8740f190d7d126e70e8bc5
I20240501 08:11:32.317413 6369 log_verifier.cc:126] Checking tablet 532f21629fae48408e569bdcd04c2cbd
I20240501 08:11:32.326721 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 532f21629fae48408e569bdcd04c2cbd
I20240501 08:11:32.326809 6369 log_verifier.cc:126] Checking tablet 80636827cb8c427c8317f676d8790c70
I20240501 08:11:32.336439 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 80636827cb8c427c8317f676d8790c70
I20240501 08:11:32.336527 6369 log_verifier.cc:126] Checking tablet 833a0a7dc6a047b19808085569e89368
I20240501 08:11:32.346427 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 833a0a7dc6a047b19808085569e89368
I20240501 08:11:32.346515 6369 log_verifier.cc:126] Checking tablet 9e92c6c0f9114282b832ecaddffea60a
I20240501 08:11:32.355592 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9e92c6c0f9114282b832ecaddffea60a
I20240501 08:11:32.355681 6369 log_verifier.cc:126] Checking tablet d227b998061e46f59653167740f87fd0
I20240501 08:11:32.366268 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d227b998061e46f59653167740f87fd0
I20240501 08:11:32.366371 6369 log_verifier.cc:126] Checking tablet d71f0f9eeffd476ebd70487ff9a93d7c
I20240501 08:11:32.376281 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d71f0f9eeffd476ebd70487ff9a93d7c
I20240501 08:11:32.376415 6369 log_verifier.cc:126] Checking tablet e0ebe3885c6f4e3794bb61abf33dbb97
I20240501 08:11:32.386595 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e0ebe3885c6f4e3794bb61abf33dbb97
I20240501 08:11:32.386698 6369 log_verifier.cc:126] Checking tablet e836061c829848caa03078113b94c44e
I20240501 08:11:32.396310 6369 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e836061c829848caa03078113b94c44e
I20240501 08:11:32.396415 6369 log_verifier.cc:126] Checking tablet e9cfe3dfff4848aabb0c7b420717130c
I20240501 08:11:32.406384 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet e9cfe3dfff4848aabb0c7b420717130c
I20240501 08:11:32.406484 6369 log_verifier.cc:126] Checking tablet fe8bf89936b04faca8c7ab15ce402e8f
I20240501 08:11:32.422902 6369 log_verifier.cc:177] Verified matching terms for 6 ops in tablet fe8bf89936b04faca8c7ab15ce402e8f
I20240501 08:11:32.423990 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 8322
I20240501 08:11:32.425535 8443 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:32.632948 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 8322
I20240501 08:11:32.667244 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 8453
I20240501 08:11:32.668246 8575 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:32.864888 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 8453
I20240501 08:11:32.897568 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 9560
I20240501 08:11:32.904424 9683 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:33.079706 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 9560
I20240501 08:11:33.109283 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 9696
I20240501 08:11:33.115490 9823 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:33.296162 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 9696
I20240501 08:11:33.325471 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 9832
I20240501 08:11:33.331843 9953 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:33.511976 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 9832
I20240501 08:11:33.541937 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 9962
I20240501 08:11:33.548434 10084 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:33.736831 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 9962
I20240501 08:11:33.765846 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 10092
I20240501 08:11:33.771801 10214 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:33.946719 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 10092
I20240501 08:11:33.975769 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 12083
I20240501 08:11:33.981536 12204 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:34.092916 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 12083
I20240501 08:11:34.113111 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 8232
I20240501 08:11:34.114153 8293 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:34.338378 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 8232
2024-05-01T08:11:34Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/0 (43353 ms)
[----------] 1 test from RebalanceParamTest (43354 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0
I20240501 08:11:34.418442 6369 test_util.cc:268] Using random seed: 149855216
I20240501 08:11:34.420236 6369 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:11:34.420339 6369 ts_itest-base.cc:113] --------------
I20240501 08:11:34.420388 6369 ts_itest-base.cc:114] 8 tablet servers
I20240501 08:11:34.420426 6369 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:11:34.420471 6369 ts_itest-base.cc:116] --------------
2024-05-01T08:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:11:34Z Disabled control of system clock
I20240501 08:11:34.450852 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:44861
--webserver_interface=127.6.56.126
--webserver_port=0
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.56.126:44861
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 08:11:34.775421 12802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:34.775781 12802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:34.775940 12802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:34.783285 12802 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:34.783435 12802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:11:34.783546 12802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:34.783629 12802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:11:34.783701 12802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:11:34.792748 12802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.56.126:44861
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:44861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.56.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:34.794034 12802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:34.795733 12802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:34.806792 12807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:34.807458 12808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:34.808516 12810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:34.809214 12802 server_base.cc:1034] running on GCE node
I20240501 08:11:34.809931 12802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:34.811280 12802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:34.812472 12802 hybrid_clock.cc:648] HybridClock initialized: now 1714551094812425 us; error 39 us; skew 500 ppm
I20240501 08:11:34.812780 12802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:34.814898 12802 webserver.cc:460] Webserver started at http://127.6.56.126:40883/ using document root <none> and password file <none>
I20240501 08:11:34.815609 12802 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:34.815722 12802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:34.816016 12802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:34.818665 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c48be6e775954c7093aaf62e68805ec0"
format_stamp: "Formatted at 2024-05-01 08:11:34 on dist-test-slave-f1nk"
I20240501 08:11:34.819449 12802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c48be6e775954c7093aaf62e68805ec0"
format_stamp: "Formatted at 2024-05-01 08:11:34 on dist-test-slave-f1nk"
I20240501 08:11:34.824622 12802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240501 08:11:34.827908 12816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:34.829231 12802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:34.829398 12802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c48be6e775954c7093aaf62e68805ec0"
format_stamp: "Formatted at 2024-05-01 08:11:34 on dist-test-slave-f1nk"
I20240501 08:11:34.829593 12802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:34.844882 12802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:34.845738 12802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:34.846010 12802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:34.863668 12802 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.126:44861
I20240501 08:11:34.863675 12867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.126:44861 every 8 connection(s)
I20240501 08:11:34.865028 12802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:11:34.868728 12868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:34.870018 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 12802
I20240501 08:11:34.870206 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:11:34.880776 12868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Bootstrap starting.
I20240501 08:11:34.884685 12868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:34.885718 12868 log.cc:824] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:34.888159 12868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: No bootstrap required, opened a new log
I20240501 08:11:34.894783 12868 raft_consensus.cc:348] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48be6e775954c7093aaf62e68805ec0" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 44861 } }
I20240501 08:11:34.895154 12868 raft_consensus.cc:374] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:34.895253 12868 raft_consensus.cc:729] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c48be6e775954c7093aaf62e68805ec0, State: Initialized, Role: FOLLOWER
I20240501 08:11:34.895835 12868 consensus_queue.cc:260] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48be6e775954c7093aaf62e68805ec0" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 44861 } }
I20240501 08:11:34.896041 12868 raft_consensus.cc:388] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:11:34.896163 12868 raft_consensus.cc:482] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:11:34.896346 12868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:34.898393 12868 raft_consensus.cc:504] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48be6e775954c7093aaf62e68805ec0" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 44861 } }
I20240501 08:11:34.898877 12868 leader_election.cc:302] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c48be6e775954c7093aaf62e68805ec0; no voters:
I20240501 08:11:34.899474 12868 leader_election.cc:288] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:11:34.899614 12873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:34.900251 12873 raft_consensus.cc:686] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [term 1 LEADER]: Becoming Leader. State: Replica: c48be6e775954c7093aaf62e68805ec0, State: Running, Role: LEADER
I20240501 08:11:34.900840 12873 consensus_queue.cc:237] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48be6e775954c7093aaf62e68805ec0" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 44861 } }
I20240501 08:11:34.901604 12868 sys_catalog.cc:564] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:11:34.903698 12874 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c48be6e775954c7093aaf62e68805ec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48be6e775954c7093aaf62e68805ec0" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 44861 } } }
I20240501 08:11:34.903949 12874 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:34.904587 12875 sys_catalog.cc:455] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c48be6e775954c7093aaf62e68805ec0. Latest consensus state: current_term: 1 leader_uuid: "c48be6e775954c7093aaf62e68805ec0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c48be6e775954c7093aaf62e68805ec0" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 44861 } } }
I20240501 08:11:34.904780 12879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:11:34.904911 12875 sys_catalog.cc:458] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0 [sys.catalog]: This master's current role is: LEADER
I20240501 08:11:34.909955 12879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:11:34.919415 12879 catalog_manager.cc:1347] Generated new cluster ID: f810deffb4eb4623aa4624b54f841050
I20240501 08:11:34.919534 12879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:11:34.956108 12879 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:11:34.957310 12879 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:11:34.967870 12879 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Generated new TSK 0
I20240501 08:11:34.968945 12879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:11:34.979221 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.65:0
--local_ip_for_outbound_sockets=127.6.56.65
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:11:35.316051 12892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:35.316403 12892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:35.316512 12892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:35.324180 12892 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:35.324360 12892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:35.324519 12892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.65
I20240501 08:11:35.334324 12892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.65
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:35.335718 12892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:35.337574 12892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:35.349900 12897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:35.350708 12898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:35.351363 12900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:35.352499 12892 server_base.cc:1034] running on GCE node
I20240501 08:11:35.353106 12892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:35.354079 12892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:35.355283 12892 hybrid_clock.cc:648] HybridClock initialized: now 1714551095355235 us; error 44 us; skew 500 ppm
I20240501 08:11:35.355594 12892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:35.357936 12892 webserver.cc:460] Webserver started at http://127.6.56.65:45057/ using document root <none> and password file <none>
I20240501 08:11:35.358709 12892 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:35.358839 12892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:35.359186 12892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:35.361917 12892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "76d830e199eb461c8ca689c9460ee0c2"
format_stamp: "Formatted at 2024-05-01 08:11:35 on dist-test-slave-f1nk"
I20240501 08:11:35.362731 12892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "76d830e199eb461c8ca689c9460ee0c2"
format_stamp: "Formatted at 2024-05-01 08:11:35 on dist-test-slave-f1nk"
I20240501 08:11:35.367909 12892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:11:35.371398 12906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:35.372802 12892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:35.372960 12892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "76d830e199eb461c8ca689c9460ee0c2"
format_stamp: "Formatted at 2024-05-01 08:11:35 on dist-test-slave-f1nk"
I20240501 08:11:35.373160 12892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:35.399041 12892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:35.399976 12892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:35.400265 12892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:35.401352 12892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:35.402958 12892 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:35.403079 12892 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:35.403188 12892 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:35.403249 12892 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:35.434798 12892 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.65:46839
I20240501 08:11:35.434849 13018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.65:46839 every 8 connection(s)
I20240501 08:11:35.436142 12892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:11:35.440837 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 12892
I20240501 08:11:35.441092 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:11:35.445348 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.66:0
--local_ip_for_outbound_sockets=127.6.56.66
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:35.451121 13019 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:11:35.451496 13019 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:35.452567 13019 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:11:35.454970 12833 ts_manager.cc:194] Registered new tserver with Master: 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:35.457224 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.65:45285
W20240501 08:11:35.776540 13023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:35.776877 13023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:35.777000 13023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:35.784575 13023 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:35.784703 13023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:35.784822 13023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.66
I20240501 08:11:35.793745 13023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.66
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:35.795063 13023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:35.796839 13023 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:35.809108 13028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:35.809756 13029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:35.810773 13031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:35.811728 13023 server_base.cc:1034] running on GCE node
I20240501 08:11:35.812371 13023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:35.813282 13023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:35.814476 13023 hybrid_clock.cc:648] HybridClock initialized: now 1714551095814428 us; error 42 us; skew 500 ppm
I20240501 08:11:35.814792 13023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:35.822237 13023 webserver.cc:460] Webserver started at http://127.6.56.66:36527/ using document root <none> and password file <none>
I20240501 08:11:35.823050 13023 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:35.823163 13023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:35.823489 13023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:35.826196 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e3fefd4dca2d41678187047f06c10187"
format_stamp: "Formatted at 2024-05-01 08:11:35 on dist-test-slave-f1nk"
I20240501 08:11:35.827004 13023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e3fefd4dca2d41678187047f06c10187"
format_stamp: "Formatted at 2024-05-01 08:11:35 on dist-test-slave-f1nk"
I20240501 08:11:35.832518 13023 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 08:11:35.836217 13037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:35.837611 13023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:35.837772 13023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e3fefd4dca2d41678187047f06c10187"
format_stamp: "Formatted at 2024-05-01 08:11:35 on dist-test-slave-f1nk"
I20240501 08:11:35.837980 13023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:35.866293 13023 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:35.867368 13023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:35.867724 13023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:35.868919 13023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:35.870787 13023 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:35.870893 13023 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:35.871003 13023 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:35.871083 13023 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:35.903920 13023 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.66:35509
I20240501 08:11:35.903977 13149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.66:35509 every 8 connection(s)
I20240501 08:11:35.905329 13023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:11:35.909945 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 13023
I20240501 08:11:35.910143 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:11:35.913689 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.67:0
--local_ip_for_outbound_sockets=127.6.56.67
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:35.919450 13150 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:11:35.919787 13150 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:35.920650 13150 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:11:35.921973 12833 ts_manager.cc:194] Registered new tserver with Master: e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:35.922756 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.66:33525
W20240501 08:11:36.249030 13153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:36.249349 13153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:36.249500 13153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:36.256879 13153 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:36.257028 13153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:36.257197 13153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.67
I20240501 08:11:36.266106 13153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.67
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:36.267482 13153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:36.269313 13153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:36.281528 13158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:36.282302 13159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:36.283265 13161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:36.284183 13153 server_base.cc:1034] running on GCE node
I20240501 08:11:36.284977 13153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:36.285914 13153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:36.287122 13153 hybrid_clock.cc:648] HybridClock initialized: now 1714551096287077 us; error 45 us; skew 500 ppm
I20240501 08:11:36.287451 13153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:36.289772 13153 webserver.cc:460] Webserver started at http://127.6.56.67:35199/ using document root <none> and password file <none>
I20240501 08:11:36.290536 13153 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:36.290658 13153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:36.290968 13153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:36.293681 13153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "6ed448d04e4646658c0de37d829ff15b"
format_stamp: "Formatted at 2024-05-01 08:11:36 on dist-test-slave-f1nk"
I20240501 08:11:36.294489 13153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "6ed448d04e4646658c0de37d829ff15b"
format_stamp: "Formatted at 2024-05-01 08:11:36 on dist-test-slave-f1nk"
I20240501 08:11:36.299868 13153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 08:11:36.303390 13167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:36.304947 13153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240501 08:11:36.305121 13153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "6ed448d04e4646658c0de37d829ff15b"
format_stamp: "Formatted at 2024-05-01 08:11:36 on dist-test-slave-f1nk"
I20240501 08:11:36.305332 13153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:36.344902 13153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:36.345809 13153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:36.346088 13153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:36.347071 13153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:36.348627 13153 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:36.348727 13153 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:36.348845 13153 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:36.348917 13153 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:36.380493 13153 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.67:33645
I20240501 08:11:36.380563 13279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.67:33645 every 8 connection(s)
I20240501 08:11:36.381786 13153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:11:36.386902 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 13153
I20240501 08:11:36.387080 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:11:36.390129 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.68:0
--local_ip_for_outbound_sockets=127.6.56.68
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:36.395097 13280 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:11:36.395414 13280 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:36.396283 13280 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:11:36.398025 12833 ts_manager.cc:194] Registered new tserver with Master: 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:36.399219 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.67:38385
I20240501 08:11:36.460758 13019 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
W20240501 08:11:36.732493 13283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:36.732800 13283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:36.732923 13283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:36.740646 13283 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:36.740792 13283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:36.740952 13283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.68
I20240501 08:11:36.750675 13283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.68
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:36.752059 13283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:36.753865 13283 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:36.766185 13288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:36.766911 13291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:36.767779 13289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:36.768637 13283 server_base.cc:1034] running on GCE node
I20240501 08:11:36.769239 13283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:36.770162 13283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:36.771361 13283 hybrid_clock.cc:648] HybridClock initialized: now 1714551096771314 us; error 41 us; skew 500 ppm
I20240501 08:11:36.771677 13283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:36.774051 13283 webserver.cc:460] Webserver started at http://127.6.56.68:41447/ using document root <none> and password file <none>
I20240501 08:11:36.774803 13283 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:36.774905 13283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:36.775208 13283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:36.777884 13283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "4ecb724cb4764a53988fde00407db4c0"
format_stamp: "Formatted at 2024-05-01 08:11:36 on dist-test-slave-f1nk"
I20240501 08:11:36.778679 13283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "4ecb724cb4764a53988fde00407db4c0"
format_stamp: "Formatted at 2024-05-01 08:11:36 on dist-test-slave-f1nk"
I20240501 08:11:36.783747 13283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240501 08:11:36.787289 13297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:36.788815 13283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:11:36.788973 13283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "4ecb724cb4764a53988fde00407db4c0"
format_stamp: "Formatted at 2024-05-01 08:11:36 on dist-test-slave-f1nk"
I20240501 08:11:36.789180 13283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:36.812767 13283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:36.813668 13283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:36.813930 13283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:36.814898 13283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:36.816520 13283 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:36.816618 13283 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:36.816725 13283 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:36.816793 13283 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:36.848083 13283 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.68:36991
I20240501 08:11:36.848134 13409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.68:36991 every 8 connection(s)
I20240501 08:11:36.849428 13283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:11:36.854321 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 13283
I20240501 08:11:36.854483 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:11:36.857795 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.69:0
--local_ip_for_outbound_sockets=127.6.56.69
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:36.862933 13410 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:11:36.863320 13410 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:36.864236 13410 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:11:36.865806 12833 ts_manager.cc:194] Registered new tserver with Master: 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:36.866909 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.68:40521
I20240501 08:11:36.925189 13150 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
W20240501 08:11:37.192204 13413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:37.192574 13413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:37.192715 13413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:37.200193 13413 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:37.200362 13413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:37.200525 13413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.69
I20240501 08:11:37.209636 13413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.69
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:37.210965 13413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:37.212690 13413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:37.224824 13418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:37.225754 13421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:37.226612 13419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:37.227442 13413 server_base.cc:1034] running on GCE node
I20240501 08:11:37.228122 13413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:37.229125 13413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:37.230346 13413 hybrid_clock.cc:648] HybridClock initialized: now 1714551097230282 us; error 53 us; skew 500 ppm
I20240501 08:11:37.230669 13413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:37.233001 13413 webserver.cc:460] Webserver started at http://127.6.56.69:34933/ using document root <none> and password file <none>
I20240501 08:11:37.233819 13413 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:37.233924 13413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:37.234256 13413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:37.239517 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "982f49fc87f149439cda0de2261b6bfd"
format_stamp: "Formatted at 2024-05-01 08:11:37 on dist-test-slave-f1nk"
I20240501 08:11:37.240414 13413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "982f49fc87f149439cda0de2261b6bfd"
format_stamp: "Formatted at 2024-05-01 08:11:37 on dist-test-slave-f1nk"
I20240501 08:11:37.245838 13413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240501 08:11:37.249408 13427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:37.250838 13413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:11:37.251008 13413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "982f49fc87f149439cda0de2261b6bfd"
format_stamp: "Formatted at 2024-05-01 08:11:37 on dist-test-slave-f1nk"
I20240501 08:11:37.251204 13413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:37.272264 13413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:37.273205 13413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:37.273486 13413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:37.274503 13413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:37.276007 13413 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:37.276108 13413 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:37.276222 13413 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:37.276315 13413 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:37.308233 13413 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.69:42897
I20240501 08:11:37.308277 13541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.69:42897 every 8 connection(s)
I20240501 08:11:37.309674 13413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:11:37.312060 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 13413
I20240501 08:11:37.312268 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:11:37.315708 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.70:0
--local_ip_for_outbound_sockets=127.6.56.70
--webserver_interface=127.6.56.70
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:37.325217 13542 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:11:37.325538 13542 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:37.326301 13542 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:11:37.327684 12833 ts_manager.cc:194] Registered new tserver with Master: 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:11:37.328492 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.69:42347
I20240501 08:11:37.402436 13280 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
W20240501 08:11:37.655175 13546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:37.655531 13546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:37.655678 13546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:37.663364 13546 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:37.663498 13546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:37.663668 13546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.70
I20240501 08:11:37.672900 13546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.56.70
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.70
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:37.674283 13546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:37.676128 13546 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:37.688624 13551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:37.689461 13552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:37.690371 13554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:37.691183 13546 server_base.cc:1034] running on GCE node
I20240501 08:11:37.691802 13546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:37.692831 13546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:37.694029 13546 hybrid_clock.cc:648] HybridClock initialized: now 1714551097693974 us; error 56 us; skew 500 ppm
I20240501 08:11:37.694355 13546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:37.696573 13546 webserver.cc:460] Webserver started at http://127.6.56.70:34153/ using document root <none> and password file <none>
I20240501 08:11:37.697305 13546 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:37.697418 13546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:37.697722 13546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:37.700275 13546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "4d516916eeac468db514eeb928f06190"
format_stamp: "Formatted at 2024-05-01 08:11:37 on dist-test-slave-f1nk"
I20240501 08:11:37.701092 13546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "4d516916eeac468db514eeb928f06190"
format_stamp: "Formatted at 2024-05-01 08:11:37 on dist-test-slave-f1nk"
I20240501 08:11:37.706308 13546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 08:11:37.709867 13560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:37.711357 13546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240501 08:11:37.711524 13546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4d516916eeac468db514eeb928f06190"
format_stamp: "Formatted at 2024-05-01 08:11:37 on dist-test-slave-f1nk"
I20240501 08:11:37.711737 13546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:37.725636 13546 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:37.726495 13546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:37.726761 13546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:37.727748 13546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:37.729286 13546 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:37.729388 13546 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:37.729511 13546 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:37.729585 13546 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:37.761700 13546 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.70:43301
I20240501 08:11:37.761755 13672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.70:43301 every 8 connection(s)
I20240501 08:11:37.763156 13546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:11:37.768746 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 13546
I20240501 08:11:37.768901 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:11:37.771874 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.71:0
--local_ip_for_outbound_sockets=127.6.56.71
--webserver_interface=127.6.56.71
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:37.776525 13673 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:11:37.776899 13673 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:37.777660 13673 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:11:37.779093 12833 ts_manager.cc:194] Registered new tserver with Master: 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:11:37.779891 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.70:52215
I20240501 08:11:37.870047 13410 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
W20240501 08:11:38.103225 13676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:38.103534 13676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:38.103652 13676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:38.111212 13676 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:38.111377 13676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:38.111546 13676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.71
I20240501 08:11:38.120901 13676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.56.71
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.71
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:38.122222 13676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:38.123968 13676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:38.137017 13682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:38.137617 13683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:38.138415 13685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:38.139358 13676 server_base.cc:1034] running on GCE node
I20240501 08:11:38.139955 13676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:38.140938 13676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:38.142169 13676 hybrid_clock.cc:648] HybridClock initialized: now 1714551098142123 us; error 54 us; skew 500 ppm
I20240501 08:11:38.142500 13676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:38.144896 13676 webserver.cc:460] Webserver started at http://127.6.56.71:33273/ using document root <none> and password file <none>
I20240501 08:11:38.145653 13676 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:38.145761 13676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:38.146075 13676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:38.148756 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "5dd722c770d34d6c9154a87917117161"
format_stamp: "Formatted at 2024-05-01 08:11:38 on dist-test-slave-f1nk"
I20240501 08:11:38.149533 13676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "5dd722c770d34d6c9154a87917117161"
format_stamp: "Formatted at 2024-05-01 08:11:38 on dist-test-slave-f1nk"
I20240501 08:11:38.154919 13676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:11:38.158684 13691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:38.160102 13676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:38.160269 13676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5dd722c770d34d6c9154a87917117161"
format_stamp: "Formatted at 2024-05-01 08:11:38 on dist-test-slave-f1nk"
I20240501 08:11:38.160521 13676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:38.178702 13676 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:38.179592 13676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:38.179863 13676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:38.180866 13676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:38.182333 13676 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:38.182421 13676 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:38.182543 13676 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:38.182613 13676 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:38.214941 13676 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.71:36997
I20240501 08:11:38.215018 13803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.71:36997 every 8 connection(s)
I20240501 08:11:38.216408 13676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:11:38.224481 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 13676
I20240501 08:11:38.224682 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:11:38.227939 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.72:0
--local_ip_for_outbound_sockets=127.6.56.72
--webserver_interface=127.6.56.72
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:11:38.228788 13804 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:11:38.229087 13804 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:38.229861 13804 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:11:38.231191 12833 ts_manager.cc:194] Registered new tserver with Master: 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:11:38.231958 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.71:60911
I20240501 08:11:38.331133 13542 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
W20240501 08:11:38.566852 13807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:38.567183 13807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:11:38.567327 13807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:38.574836 13807 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:11:38.574996 13807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:11:38.575155 13807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.72
I20240501 08:11:38.584381 13807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.56.72
--webserver_port=0
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.72
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:11:38.585741 13807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:11:38.587469 13807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:11:38.600124 13813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:38.600613 13814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:11:38.601451 13816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:11:38.602241 13807 server_base.cc:1034] running on GCE node
I20240501 08:11:38.602927 13807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:11:38.603945 13807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:11:38.605172 13807 hybrid_clock.cc:648] HybridClock initialized: now 1714551098605126 us; error 44 us; skew 500 ppm
I20240501 08:11:38.605506 13807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:11:38.607865 13807 webserver.cc:460] Webserver started at http://127.6.56.72:46621/ using document root <none> and password file <none>
I20240501 08:11:38.608664 13807 fs_manager.cc:359] Metadata directory not provided
I20240501 08:11:38.608773 13807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:11:38.609088 13807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:11:38.611969 13807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
format_stamp: "Formatted at 2024-05-01 08:11:38 on dist-test-slave-f1nk"
I20240501 08:11:38.612811 13807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
format_stamp: "Formatted at 2024-05-01 08:11:38 on dist-test-slave-f1nk"
I20240501 08:11:38.618104 13807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240501 08:11:38.621624 13822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:38.623087 13807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:11:38.623242 13807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
format_stamp: "Formatted at 2024-05-01 08:11:38 on dist-test-slave-f1nk"
I20240501 08:11:38.623440 13807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:11:38.648695 13807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:11:38.649587 13807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:11:38.649876 13807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:11:38.650837 13807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:11:38.652359 13807 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:11:38.652451 13807 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:38.652567 13807 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:11:38.652639 13807 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:11:38.684788 13807 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.72:45383
I20240501 08:11:38.684834 13934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.72:45383 every 8 connection(s)
I20240501 08:11:38.686089 13807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:11:38.688689 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 13807
I20240501 08:11:38.688887 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:11:38.698868 13935 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:11:38.699165 13935 heartbeater.cc:464] Registering TS with master...
I20240501 08:11:38.699862 13935 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:11:38.701165 12833 ts_manager.cc:194] Registered new tserver with Master: 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:11:38.701972 12833 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.72:33439
I20240501 08:11:38.705066 6369 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:11:38.724509 6369 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:11:38.724678 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 13413
I20240501 08:11:38.732501 13537 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:38.783026 13673 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
I20240501 08:11:38.813973 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 13413
I20240501 08:11:38.830046 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 13546
I20240501 08:11:38.838131 13668 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:38.920943 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 13546
I20240501 08:11:38.937150 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 13676
I20240501 08:11:38.945144 13799 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:39.025616 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 13676
I20240501 08:11:39.041481 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 13807
I20240501 08:11:39.049947 13930 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:11:39.131361 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 13807
I20240501 08:11:42.899426 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:42.955582 12952 tablet_service.cc:1467] Processing CreateTablet for tablet baa03907546a4297a8eba0710a6b7acc (DEFAULT_TABLE table=rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:42.955183 13345 tablet_service.cc:1467] Processing CreateTablet for tablet ac31ed51cbd2474cb93c5393358848ec (DEFAULT_TABLE table=rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:42.955601 13085 tablet_service.cc:1467] Processing CreateTablet for tablet e89f532aaa2f4e8e8155c1fc9b4767cc (DEFAULT_TABLE table=rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:42.956141 12954 tablet_service.cc:1467] Processing CreateTablet for tablet e89f532aaa2f4e8e8155c1fc9b4767cc (DEFAULT_TABLE table=rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:42.955183 12953 tablet_service.cc:1467] Processing CreateTablet for tablet ac31ed51cbd2474cb93c5393358848ec (DEFAULT_TABLE table=rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:42.956895 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89f532aaa2f4e8e8155c1fc9b4767cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:42.956898 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa03907546a4297a8eba0710a6b7acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:42.956904 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac31ed51cbd2474cb93c5393358848ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:42.957346 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac31ed51cbd2474cb93c5393358848ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:42.959560 13344 tablet_service.cc:1467] Processing CreateTablet for tablet baa03907546a4297a8eba0710a6b7acc (DEFAULT_TABLE table=rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:42.960081 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa03907546a4297a8eba0710a6b7acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:42.960455 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89f532aaa2f4e8e8155c1fc9b4767cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:42.961349 13084 tablet_service.cc:1467] Processing CreateTablet for tablet ac31ed51cbd2474cb93c5393358848ec (DEFAULT_TABLE table=rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:42.961869 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac31ed51cbd2474cb93c5393358848ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:42.975226 13965 tablet_bootstrap.cc:492] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:42.980796 13967 tablet_bootstrap.cc:492] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:42.980901 13966 tablet_bootstrap.cc:492] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:42.985368 13966 tablet_bootstrap.cc:654] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:42.985392 13215 tablet_service.cc:1467] Processing CreateTablet for tablet e89f532aaa2f4e8e8155c1fc9b4767cc (DEFAULT_TABLE table=rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:42.986066 13214 tablet_service.cc:1467] Processing CreateTablet for tablet baa03907546a4297a8eba0710a6b7acc (DEFAULT_TABLE table=rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:42.986755 13965 tablet_bootstrap.cc:654] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:42.986725 13966 log.cc:824] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:42.987244 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa03907546a4297a8eba0710a6b7acc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:42.987788 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89f532aaa2f4e8e8155c1fc9b4767cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:42.988257 13965 log.cc:824] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:42.989761 13967 tablet_bootstrap.cc:654] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:42.991533 13967 log.cc:824] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:42.996937 13966 tablet_bootstrap.cc:492] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:42.997310 13965 tablet_bootstrap.cc:492] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:42.997310 13966 ts_tablet_manager.cc:1367] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
I20240501 08:11:42.997611 13965 ts_tablet_manager.cc:1367] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.000s
I20240501 08:11:43.005048 13967 tablet_bootstrap.cc:492] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:43.005411 13967 ts_tablet_manager.cc:1367] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.007s
I20240501 08:11:43.006945 13966 raft_consensus.cc:348] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.007447 13966 raft_consensus.cc:374] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.007563 13966 raft_consensus.cc:729] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.008621 13966 consensus_queue.cc:260] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.011392 13971 tablet_bootstrap.cc:492] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:43.015904 13965 raft_consensus.cc:348] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.016453 13965 raft_consensus.cc:374] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.016558 13965 raft_consensus.cc:729] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.017213 13966 ts_tablet_manager.cc:1398] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.020s user 0.018s sys 0.001s
I20240501 08:11:43.017344 13965 consensus_queue.cc:260] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.017690 13966 tablet_bootstrap.cc:492] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:43.017459 13967 raft_consensus.cc:348] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.017971 13967 raft_consensus.cc:374] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.018088 13967 raft_consensus.cc:729] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.019371 13967 consensus_queue.cc:260] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.020193 13966 tablet_bootstrap.cc:654] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.025995 13965 ts_tablet_manager.cc:1398] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.028s user 0.013s sys 0.009s
I20240501 08:11:43.026486 13965 tablet_bootstrap.cc:492] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:43.026486 13966 tablet_bootstrap.cc:492] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:43.026643 13966 ts_tablet_manager.cc:1367] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 08:11:43.027825 13971 tablet_bootstrap.cc:654] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.027743 13966 raft_consensus.cc:348] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.027981 13966 raft_consensus.cc:374] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.028071 13966 raft_consensus.cc:729] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.028398 13966 consensus_queue.cc:260] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.028671 13965 tablet_bootstrap.cc:654] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.029137 13966 ts_tablet_manager.cc:1398] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:43.030984 13965 tablet_bootstrap.cc:492] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:43.031157 13965 ts_tablet_manager.cc:1367] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:43.032095 13971 log.cc:824] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: Log is configured to *not* fsync() on all Append() calls
I20240501 08:11:43.032220 13965 raft_consensus.cc:348] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.032474 13965 raft_consensus.cc:374] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.032574 13965 raft_consensus.cc:729] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.032809 13965 consensus_queue.cc:260] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.033483 13965 ts_tablet_manager.cc:1398] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:43.034096 13967 ts_tablet_manager.cc:1398] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.028s user 0.012s sys 0.010s
I20240501 08:11:43.034586 13967 tablet_bootstrap.cc:492] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:43.036487 13967 tablet_bootstrap.cc:654] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.041230 13967 tablet_bootstrap.cc:492] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:43.041401 13967 ts_tablet_manager.cc:1367] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240501 08:11:43.042217 13971 tablet_bootstrap.cc:492] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:43.042243 13967 raft_consensus.cc:348] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.042464 13967 raft_consensus.cc:374] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.042536 13971 ts_tablet_manager.cc:1367] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.004s
I20240501 08:11:43.042564 13967 raft_consensus.cc:729] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.042805 13967 consensus_queue.cc:260] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.043308 13967 ts_tablet_manager.cc:1398] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:43.043556 13967 tablet_bootstrap.cc:492] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:43.045258 13967 tablet_bootstrap.cc:654] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.047233 13967 tablet_bootstrap.cc:492] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:43.047396 13967 ts_tablet_manager.cc:1367] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:43.048110 13967 raft_consensus.cc:348] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.048362 13967 raft_consensus.cc:374] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.048447 13967 raft_consensus.cc:729] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.048694 13967 consensus_queue.cc:260] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.049196 13967 ts_tablet_manager.cc:1398] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:43.050740 13971 raft_consensus.cc:348] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.051209 13971 raft_consensus.cc:374] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.051326 13971 raft_consensus.cc:729] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.052100 13971 consensus_queue.cc:260] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.054612 13971 ts_tablet_manager.cc:1398] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20240501 08:11:43.055006 13971 tablet_bootstrap.cc:492] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:43.056797 13971 tablet_bootstrap.cc:654] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:43.058821 13971 tablet_bootstrap.cc:492] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:43.058956 13971 ts_tablet_manager.cc:1367] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:43.059774 13971 raft_consensus.cc:348] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.059947 13971 raft_consensus.cc:374] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:43.060045 13971 raft_consensus.cc:729] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:43.060253 13971 consensus_queue.cc:260] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.060751 13971 ts_tablet_manager.cc:1398] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:43.080996 13973 raft_consensus.cc:482] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:43.081306 13973 raft_consensus.cc:504] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.083220 13973 leader_election.cc:288] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:43.090188 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa03907546a4297a8eba0710a6b7acc" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:43.090190 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa03907546a4297a8eba0710a6b7acc" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:43.090723 13235 raft_consensus.cc:2454] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:43.090726 12974 raft_consensus.cc:2454] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:43.091331 13301 leader_election.cc:302] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:11:43.091714 13973 raft_consensus.cc:2781] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:43.091890 13973 raft_consensus.cc:482] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:43.092000 13973 raft_consensus.cc:3037] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.094195 13973 raft_consensus.cc:504] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.094827 13973 leader_election.cc:288] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:43.095384 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa03907546a4297a8eba0710a6b7acc" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:43.095449 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa03907546a4297a8eba0710a6b7acc" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:43.095624 12974 raft_consensus.cc:3037] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.095638 13235 raft_consensus.cc:3037] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.098901 13235 raft_consensus.cc:2454] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:43.098901 12974 raft_consensus.cc:2454] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:43.099424 13301 leader_election.cc:302] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:11:43.099761 13973 raft_consensus.cc:2781] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:43.100231 13973 raft_consensus.cc:686] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:11:43.100764 13973 consensus_queue.cc:237] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.105023 12831 catalog_manager.cc:5484] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
W20240501 08:11:43.106317 13411 tablet.cc:2369] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:11:43.138665 13281 tablet.cc:2369] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:11:43.162596 13151 tablet.cc:2369] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:11:43.193929 13020 tablet.cc:2369] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:11:43.236933 13974 raft_consensus.cc:482] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:43.237150 13974 raft_consensus.cc:504] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.238878 13974 leader_election.cc:288] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:43.239548 13973 raft_consensus.cc:482] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:43.239814 13973 raft_consensus.cc:504] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.241123 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac31ed51cbd2474cb93c5393358848ec" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:43.241222 13973 leader_election.cc:288] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:43.241413 12974 raft_consensus.cc:2454] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:43.242352 13301 leader_election.cc:302] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:43.242674 13973 raft_consensus.cc:2781] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:43.242781 13973 raft_consensus.cc:482] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:43.242928 13973 raft_consensus.cc:3037] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.246102 13973 raft_consensus.cc:504] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.247357 13973 leader_election.cc:288] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:43.248417 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac31ed51cbd2474cb93c5393358848ec" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:43.248677 12974 raft_consensus.cc:3037] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.249449 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac31ed51cbd2474cb93c5393358848ec" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:43.249934 13365 raft_consensus.cc:2379] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76d830e199eb461c8ca689c9460ee0c2 in current term 1: Already voted for candidate 4ecb724cb4764a53988fde00407db4c0 in this term.
I20240501 08:11:43.250771 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac31ed51cbd2474cb93c5393358848ec" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:43.251075 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac31ed51cbd2474cb93c5393358848ec" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:43.250926 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac31ed51cbd2474cb93c5393358848ec" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:43.251561 12974 raft_consensus.cc:2454] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:43.251693 13104 raft_consensus.cc:2454] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:43.252200 13301 leader_election.cc:302] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:43.252344 12909 leader_election.cc:302] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2; no voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187
I20240501 08:11:43.252540 13973 raft_consensus.cc:2781] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:43.252801 13974 raft_consensus.cc:2726] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:11:43.252859 13973 raft_consensus.cc:686] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:11:43.253162 13973 consensus_queue.cc:237] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:43.255712 12831 catalog_manager.cc:5484] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:11:43.287254 13980 raft_consensus.cc:482] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:43.287484 13980 raft_consensus.cc:504] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.289297 13980 leader_election.cc:288] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:43.297065 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89f532aaa2f4e8e8155c1fc9b4767cc" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:43.297366 13103 raft_consensus.cc:2454] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:43.297528 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89f532aaa2f4e8e8155c1fc9b4767cc" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:43.297767 12974 raft_consensus.cc:2454] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:43.297886 13170 leader_election.cc:302] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:43.298337 13980 raft_consensus.cc:2781] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:43.298471 13980 raft_consensus.cc:482] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:43.298565 13980 raft_consensus.cc:3037] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.300679 13980 raft_consensus.cc:504] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.301296 13980 leader_election.cc:288] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:43.301822 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89f532aaa2f4e8e8155c1fc9b4767cc" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:43.301880 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89f532aaa2f4e8e8155c1fc9b4767cc" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:43.302032 13103 raft_consensus.cc:3037] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.302083 12974 raft_consensus.cc:3037] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.304098 12974 raft_consensus.cc:2454] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:43.304142 13103 raft_consensus.cc:2454] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:43.304608 13170 leader_election.cc:302] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:43.304934 13980 raft_consensus.cc:2781] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:43.305414 13980 raft_consensus.cc:686] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:11:43.305938 13980 consensus_queue.cc:237] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:43.310416 12831 catalog_manager.cc:5484] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:43.500677 13973 consensus_queue.cc:1035] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:43.508548 13973 consensus_queue.cc:1035] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:43.686404 13103 raft_consensus.cc:3037] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:43.690701 13973 consensus_queue.cc:1035] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:43.692577 13980 consensus_queue.cc:1035] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:43.699445 13980 consensus_queue.cc:1035] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:43.700860 13973 consensus_queue.cc:1035] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:43.754587 13994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:43.754925 13994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:43.762514 13994 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 252.291 ms
time per row: 0.126146 ms
I20240501 08:11:44.162726 12832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:44.171921 13214 tablet_service.cc:1467] Processing CreateTablet for tablet a54efcd41aeb4c24adfa51bb543d9661 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.172034 13084 tablet_service.cc:1467] Processing CreateTablet for tablet a54efcd41aeb4c24adfa51bb543d9661 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.172468 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54efcd41aeb4c24adfa51bb543d9661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.172482 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54efcd41aeb4c24adfa51bb543d9661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.173094 13345 tablet_service.cc:1467] Processing CreateTablet for tablet a54efcd41aeb4c24adfa51bb543d9661 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.173235 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.173561 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.174679 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc5dbfe3a9b4c84a3217981527e73ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.175127 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc5dbfe3a9b4c84a3217981527e73ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.179793 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.180400 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.182258 13083 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc5dbfe3a9b4c84a3217981527e73ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.182868 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc5dbfe3a9b4c84a3217981527e73ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.184661 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54efcd41aeb4c24adfa51bb543d9661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.184978 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc5dbfe3a9b4c84a3217981527e73ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.185595 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc5dbfe3a9b4c84a3217981527e73ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.188958 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:44.189570 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:44.205696 14061 tablet_bootstrap.cc:492] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:44.207674 14061 tablet_bootstrap.cc:654] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.216745 14059 tablet_bootstrap.cc:492] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:44.216948 14060 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:44.217517 14058 tablet_bootstrap.cc:492] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:44.218518 14061 tablet_bootstrap.cc:492] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:44.218684 14061 ts_tablet_manager.cc:1367] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 08:11:44.219249 14060 tablet_bootstrap.cc:654] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.219617 14059 tablet_bootstrap.cc:654] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.219640 14058 tablet_bootstrap.cc:654] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.219784 14061 raft_consensus.cc:348] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:44.220021 14061 raft_consensus.cc:374] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.220117 14061 raft_consensus.cc:729] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.220508 14061 consensus_queue.cc:260] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:44.228143 14061 ts_tablet_manager.cc:1398] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240501 08:11:44.228214 14058 tablet_bootstrap.cc:492] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:44.228410 14058 ts_tablet_manager.cc:1367] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.008s
I20240501 08:11:44.228670 14060 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:44.228849 14060 ts_tablet_manager.cc:1367] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240501 08:11:44.229525 14058 raft_consensus.cc:348] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.229792 14058 raft_consensus.cc:374] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.229723 14060 raft_consensus.cc:348] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.229890 14058 raft_consensus.cc:729] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.229952 14060 raft_consensus.cc:374] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.230063 14060 raft_consensus.cc:729] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.230199 14058 consensus_queue.cc:260] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.230378 14060 consensus_queue.cc:260] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.231074 14060 ts_tablet_manager.cc:1398] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:44.231410 14060 tablet_bootstrap.cc:492] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:44.232867 14059 tablet_bootstrap.cc:492] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:44.233126 14060 tablet_bootstrap.cc:654] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.233680 14059 ts_tablet_manager.cc:1367] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240501 08:11:44.234995 14059 raft_consensus.cc:348] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.235396 14059 raft_consensus.cc:374] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.235540 14059 raft_consensus.cc:729] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.235857 14059 consensus_queue.cc:260] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.236068 14058 ts_tablet_manager.cc:1398] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20240501 08:11:44.236572 14059 ts_tablet_manager.cc:1398] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:44.236948 14058 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:44.238679 14059 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:44.238720 14058 tablet_bootstrap.cc:654] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.242568 14059 tablet_bootstrap.cc:654] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.244904 14058 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:44.244904 14060 tablet_bootstrap.cc:492] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:44.245098 14058 ts_tablet_manager.cc:1367] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240501 08:11:44.245100 14060 ts_tablet_manager.cc:1367] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20240501 08:11:44.245239 14059 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:44.245411 14059 ts_tablet_manager.cc:1367] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240501 08:11:44.245908 14058 raft_consensus.cc:348] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.245908 14060 raft_consensus.cc:348] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.245992 14059 raft_consensus.cc:348] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.246147 14058 raft_consensus.cc:374] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.246147 14060 raft_consensus.cc:374] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.246199 14059 raft_consensus.cc:374] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.246228 14060 raft_consensus.cc:729] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.246228 14058 raft_consensus.cc:729] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.246284 14059 raft_consensus.cc:729] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.246485 14058 consensus_queue.cc:260] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.246484 14060 consensus_queue.cc:260] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.246577 14059 consensus_queue.cc:260] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.247043 14060 ts_tablet_manager.cc:1398] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:44.247115 14058 ts_tablet_manager.cc:1398] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:44.247397 14058 tablet_bootstrap.cc:492] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:44.247639 14059 ts_tablet_manager.cc:1398] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:44.247928 14059 tablet_bootstrap.cc:492] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:44.249289 14058 tablet_bootstrap.cc:654] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.250416 14059 tablet_bootstrap.cc:654] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:44.252252 14058 tablet_bootstrap.cc:492] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:44.252260 14059 tablet_bootstrap.cc:492] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:44.252416 14058 ts_tablet_manager.cc:1367] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 08:11:44.252422 14059 ts_tablet_manager.cc:1367] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:44.252986 14059 raft_consensus.cc:348] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:44.252990 14058 raft_consensus.cc:348] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:44.253149 14059 raft_consensus.cc:374] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.253156 14058 raft_consensus.cc:374] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:44.253218 14059 raft_consensus.cc:729] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.253255 14058 raft_consensus.cc:729] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:44.253472 14058 consensus_queue.cc:260] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:44.253473 14059 consensus_queue.cc:260] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:44.253911 14058 ts_tablet_manager.cc:1398] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:44.254070 14059 ts_tablet_manager.cc:1398] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:44.414393 13992 raft_consensus.cc:482] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:44.414603 13992 raft_consensus.cc:504] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:44.415397 13992 leader_election.cc:288] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:44.415946 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc5dbfe3a9b4c84a3217981527e73ea" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:44.416008 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc5dbfe3a9b4c84a3217981527e73ea" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:44.416168 12973 raft_consensus.cc:2454] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:44.416313 13103 raft_consensus.cc:2454] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:44.416756 13170 leader_election.cc:302] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:44.417088 13992 raft_consensus.cc:2781] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:44.417201 13992 raft_consensus.cc:482] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:44.417289 13992 raft_consensus.cc:3037] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.419276 13992 raft_consensus.cc:504] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:44.419890 13992 leader_election.cc:288] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:44.420419 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc5dbfe3a9b4c84a3217981527e73ea" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:44.420429 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc5dbfe3a9b4c84a3217981527e73ea" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:44.420636 12973 raft_consensus.cc:3037] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.420660 13103 raft_consensus.cc:3037] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.423535 13103 raft_consensus.cc:2454] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:44.423548 12973 raft_consensus.cc:2454] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:44.423983 13170 leader_election.cc:302] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:44.424329 13992 raft_consensus.cc:2781] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:44.424486 13992 raft_consensus.cc:686] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:11:44.424769 13992 consensus_queue.cc:237] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:44.425230 14067 raft_consensus.cc:482] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:44.425386 14067 raft_consensus.cc:504] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.427227 14067 leader_election.cc:288] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:44.427944 12832 catalog_manager.cc:5484] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:11:44.435791 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ecb2ec0b3d4fd7a02896cc9a28ba27" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:44.436045 13235 raft_consensus.cc:2454] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:44.436667 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ecb2ec0b3d4fd7a02896cc9a28ba27" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:44.436764 13041 leader_election.cc:302] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:44.436894 13365 raft_consensus.cc:2454] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:44.437232 14067 raft_consensus.cc:2781] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:44.437358 14067 raft_consensus.cc:482] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:44.437459 14067 raft_consensus.cc:3037] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.440490 14067 raft_consensus.cc:504] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.441432 14067 leader_election.cc:288] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:44.441807 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ecb2ec0b3d4fd7a02896cc9a28ba27" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:44.441828 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99ecb2ec0b3d4fd7a02896cc9a28ba27" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:44.442025 13365 raft_consensus.cc:3037] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.442044 13235 raft_consensus.cc:3037] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.445022 13235 raft_consensus.cc:2454] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:44.445034 13365 raft_consensus.cc:2454] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:44.445667 13041 leader_election.cc:302] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:44.445979 14067 raft_consensus.cc:2781] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:44.446389 14067 raft_consensus.cc:686] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:44.447080 14067 consensus_queue.cc:237] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.450522 12832 catalog_manager.cc:5484] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:11:44.530681 14043 raft_consensus.cc:482] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:44.530879 14043 raft_consensus.cc:504] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.531563 14043 leader_election.cc:288] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:44.532109 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54efcd41aeb4c24adfa51bb543d9661" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:44.532095 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54efcd41aeb4c24adfa51bb543d9661" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:44.532395 13103 raft_consensus.cc:2454] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:44.532384 13235 raft_consensus.cc:2454] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:44.532845 13300 leader_election.cc:302] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:44.533155 14043 raft_consensus.cc:2781] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:44.533291 14043 raft_consensus.cc:482] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:44.533396 14043 raft_consensus.cc:3037] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.535420 14043 raft_consensus.cc:504] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.536017 14043 leader_election.cc:288] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:44.536540 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54efcd41aeb4c24adfa51bb543d9661" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:44.536621 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54efcd41aeb4c24adfa51bb543d9661" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:44.536724 13235 raft_consensus.cc:3037] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.536835 13103 raft_consensus.cc:3037] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:44.538733 13235 raft_consensus.cc:2454] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:44.538969 13103 raft_consensus.cc:2454] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:44.539194 13301 leader_election.cc:302] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:11:44.539515 14043 raft_consensus.cc:2781] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:44.539657 14043 raft_consensus.cc:686] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:11:44.539956 14043 consensus_queue.cc:237] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:44.543151 12832 catalog_manager.cc:5484] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
W20240501 08:11:44.907033 14081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:44.907373 14081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:44.915673 14081 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:11:44.963405 13992 consensus_queue.cc:1035] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:44.976572 13992 consensus_queue.cc:1035] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:45.072248 14043 consensus_queue.cc:1035] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:45.074514 13235 raft_consensus.cc:1261] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:45.075789 14102 consensus_queue.cc:1035] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:45.077435 14102 consensus_queue.cc:1035] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:45.101974 14043 consensus_queue.cc:1035] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 250.669 ms
time per row: 0.125334 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:46.249280 12826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:46.258544 13344 tablet_service.cc:1467] Processing CreateTablet for tablet d968e8c981b041bbb290058b86506636 (DEFAULT_TABLE table=rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.258731 13084 tablet_service.cc:1467] Processing CreateTablet for tablet d968e8c981b041bbb290058b86506636 (DEFAULT_TABLE table=rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.259013 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d968e8c981b041bbb290058b86506636. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.258980 12953 tablet_service.cc:1467] Processing CreateTablet for tablet d968e8c981b041bbb290058b86506636 (DEFAULT_TABLE table=rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.259155 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d968e8c981b041bbb290058b86506636. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.259366 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d968e8c981b041bbb290058b86506636. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.260452 13083 tablet_service.cc:1467] Processing CreateTablet for tablet 442304f2fddc4a80928a3b8a44ec17c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.260821 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442304f2fddc4a80928a3b8a44ec17c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.265012 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 442304f2fddc4a80928a3b8a44ec17c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.265539 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442304f2fddc4a80928a3b8a44ec17c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.265686 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 2877095508b5400e9452adf3f737179a (DEFAULT_TABLE table=rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.266032 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2877095508b5400e9452adf3f737179a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.268857 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 2877095508b5400e9452adf3f737179a (DEFAULT_TABLE table=rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.269446 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2877095508b5400e9452adf3f737179a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.273830 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 2877095508b5400e9452adf3f737179a (DEFAULT_TABLE table=rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.274256 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2877095508b5400e9452adf3f737179a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.274992 13343 tablet_service.cc:1467] Processing CreateTablet for tablet 442304f2fddc4a80928a3b8a44ec17c6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:46.275386 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442304f2fddc4a80928a3b8a44ec17c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:46.281056 14152 tablet_bootstrap.cc:492] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:46.282714 14152 tablet_bootstrap.cc:654] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.283820 14154 tablet_bootstrap.cc:492] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:46.286351 14154 tablet_bootstrap.cc:654] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.286715 14153 tablet_bootstrap.cc:492] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:46.287185 14152 tablet_bootstrap.cc:492] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:46.287325 14152 ts_tablet_manager.cc:1367] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:11:46.288100 14152 raft_consensus.cc:348] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.288378 14153 tablet_bootstrap.cc:654] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.288365 14152 raft_consensus.cc:374] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.288460 14152 raft_consensus.cc:729] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.288702 14152 consensus_queue.cc:260] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.289207 14154 tablet_bootstrap.cc:492] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:46.289211 14152 ts_tablet_manager.cc:1398] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:46.289368 14154 ts_tablet_manager.cc:1367] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 08:11:46.289569 14152 tablet_bootstrap.cc:492] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:46.290313 14154 raft_consensus.cc:348] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:46.290552 14154 raft_consensus.cc:374] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.290650 14154 raft_consensus.cc:729] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.290961 14154 consensus_queue.cc:260] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:46.291472 14152 tablet_bootstrap.cc:654] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.291790 14154 ts_tablet_manager.cc:1398] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:46.292102 14154 tablet_bootstrap.cc:492] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:46.293349 14156 tablet_bootstrap.cc:492] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:46.293848 14154 tablet_bootstrap.cc:654] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.297010 14152 tablet_bootstrap.cc:492] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:46.297233 14152 ts_tablet_manager.cc:1367] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 08:11:46.298619 14156 tablet_bootstrap.cc:654] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.298198 14152 raft_consensus.cc:348] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.299676 14152 raft_consensus.cc:374] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.299793 14152 raft_consensus.cc:729] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.300154 14152 consensus_queue.cc:260] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.300940 14153 tablet_bootstrap.cc:492] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:46.301111 14153 ts_tablet_manager.cc:1367] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20240501 08:11:46.301573 14154 tablet_bootstrap.cc:492] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:46.301896 14154 ts_tablet_manager.cc:1367] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240501 08:11:46.301903 14153 raft_consensus.cc:348] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.302119 14153 raft_consensus.cc:374] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.302219 14153 raft_consensus.cc:729] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.302497 14153 consensus_queue.cc:260] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.303630 14153 ts_tablet_manager.cc:1398] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:46.303781 14154 raft_consensus.cc:348] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.303927 14153 tablet_bootstrap.cc:492] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:46.304004 14156 tablet_bootstrap.cc:492] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:46.304103 14154 raft_consensus.cc:374] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.304172 14156 ts_tablet_manager.cc:1367] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240501 08:11:46.304270 14154 raft_consensus.cc:729] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.304633 14154 consensus_queue.cc:260] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.304960 14156 raft_consensus.cc:348] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:46.305184 14156 raft_consensus.cc:374] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.305282 14156 raft_consensus.cc:729] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.305488 14154 ts_tablet_manager.cc:1398] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:46.305528 14153 tablet_bootstrap.cc:654] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.305523 14156 consensus_queue.cc:260] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:46.305904 14154 tablet_bootstrap.cc:492] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:46.306066 14156 ts_tablet_manager.cc:1398] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:46.303292 14152 ts_tablet_manager.cc:1398] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 08:11:46.307595 14154 tablet_bootstrap.cc:654] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.310928 14154 tablet_bootstrap.cc:492] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:46.311128 14154 ts_tablet_manager.cc:1367] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 08:11:46.311159 14153 tablet_bootstrap.cc:492] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:46.311323 14153 ts_tablet_manager.cc:1367] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:11:46.311899 14154 raft_consensus.cc:348] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.312139 14154 raft_consensus.cc:374] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.312245 14154 raft_consensus.cc:729] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.312285 14153 raft_consensus.cc:348] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:46.312562 14153 raft_consensus.cc:374] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.312489 14154 consensus_queue.cc:260] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.312646 14153 raft_consensus.cc:729] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.313066 14154 ts_tablet_manager.cc:1398] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:46.313051 14153 consensus_queue.cc:260] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:46.313588 14153 ts_tablet_manager.cc:1398] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:46.313901 14153 tablet_bootstrap.cc:492] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:46.315533 14153 tablet_bootstrap.cc:654] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:46.317417 14153 tablet_bootstrap.cc:492] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:46.317559 14153 ts_tablet_manager.cc:1367] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 08:11:46.318138 14153 raft_consensus.cc:348] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.318302 14153 raft_consensus.cc:374] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:46.318397 14153 raft_consensus.cc:729] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:46.318605 14153 consensus_queue.cc:260] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.319082 14153 ts_tablet_manager.cc:1398] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:46.445541 14109 raft_consensus.cc:482] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:46.445745 14109 raft_consensus.cc:504] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:46.446635 14109 leader_election.cc:288] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:46.447118 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2877095508b5400e9452adf3f737179a" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:46.447453 12974 raft_consensus.cc:2454] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:46.447999 13171 leader_election.cc:302] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:46.448344 14109 raft_consensus.cc:2781] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:46.448473 14109 raft_consensus.cc:482] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:46.448570 14109 raft_consensus.cc:3037] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.450675 14109 raft_consensus.cc:504] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:46.451244 14109 leader_election.cc:288] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:46.451781 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2877095508b5400e9452adf3f737179a" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:46.452020 12974 raft_consensus.cc:3037] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.453902 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2877095508b5400e9452adf3f737179a" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:46.454052 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2877095508b5400e9452adf3f737179a" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:46.454138 13365 raft_consensus.cc:2454] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:46.454267 13364 raft_consensus.cc:3037] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.454627 12974 raft_consensus.cc:2454] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:46.455021 13171 leader_election.cc:302] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:46.455356 14109 raft_consensus.cc:2781] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:46.455724 14109 raft_consensus.cc:686] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:11:46.456126 14109 consensus_queue.cc:237] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:46.456900 13364 raft_consensus.cc:2454] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:46.460336 12832 catalog_manager.cc:5484] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:46.478763 14136 raft_consensus.cc:482] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:46.478972 14136 raft_consensus.cc:504] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.479779 14136 leader_election.cc:288] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:46.480381 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d968e8c981b041bbb290058b86506636" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:46.480702 13364 raft_consensus.cc:2454] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:46.481307 13040 leader_election.cc:302] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:46.481678 14136 raft_consensus.cc:2781] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:46.481786 14136 raft_consensus.cc:482] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:46.481870 14136 raft_consensus.cc:3037] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.484474 14136 raft_consensus.cc:504] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.485286 14136 leader_election.cc:288] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:46.485400 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d968e8c981b041bbb290058b86506636" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:46.485603 13364 raft_consensus.cc:3037] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.487479 13364 raft_consensus.cc:2454] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:46.487864 13040 leader_election.cc:302] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:46.488162 14136 raft_consensus.cc:2781] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:46.488598 14136 raft_consensus.cc:686] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:46.488889 14136 consensus_queue.cc:237] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.488992 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d968e8c981b041bbb290058b86506636" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:46.489039 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d968e8c981b041bbb290058b86506636" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:46.489203 12974 raft_consensus.cc:2454] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:46.491511 12832 catalog_manager.cc:5484] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:11:46.493723 14136 raft_consensus.cc:482] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:46.493947 14136 raft_consensus.cc:504] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.494590 14136 leader_election.cc:288] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:46.494987 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442304f2fddc4a80928a3b8a44ec17c6" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:46.495049 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442304f2fddc4a80928a3b8a44ec17c6" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:46.495179 13364 raft_consensus.cc:2454] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:46.495262 12973 raft_consensus.cc:2454] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:46.495568 13040 leader_election.cc:302] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:46.495879 14136 raft_consensus.cc:2781] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:46.495956 14136 raft_consensus.cc:482] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:46.496031 14136 raft_consensus.cc:3037] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.497809 14136 raft_consensus.cc:504] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.498329 14136 leader_election.cc:288] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:46.498764 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442304f2fddc4a80928a3b8a44ec17c6" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:46.498751 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442304f2fddc4a80928a3b8a44ec17c6" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:46.498953 12973 raft_consensus.cc:3037] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.498953 13364 raft_consensus.cc:3037] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.501794 13364 raft_consensus.cc:2454] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:46.501794 12973 raft_consensus.cc:2454] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:46.502226 13041 leader_election.cc:302] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:46.502615 14136 raft_consensus.cc:2781] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:46.502755 14136 raft_consensus.cc:686] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:46.502995 14136 consensus_queue.cc:237] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:46.505656 12832 catalog_manager.cc:5484] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:11:46.840739 14169 consensus_queue.cc:1035] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:46.846269 14169 consensus_queue.cc:1035] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:46.884773 14176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:46.885221 14176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:46.893611 14176 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:46.910106 12973 raft_consensus.cc:3037] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:46.915269 14136 consensus_queue.cc:1035] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:46.932139 14136 consensus_queue.cc:1035] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240501 08:11:47.053857 12974 raft_consensus.cc:1261] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:47.056396 14173 consensus_queue.cc:1035] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:47.059448 13364 raft_consensus.cc:1261] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:47.098392 14220 consensus_queue.cc:1035] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:47.102931 13041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 46826.7us
I20240501 08:11:47.155735 14194 mvcc.cc:205] Tried to move back new op lower bound from 7022801334523289600 to 7022801332238249984. Current Snapshot: MvccSnapshot[applied={T|T < 7022801334479527936}]
I20240501 08:11:47.176403 14182 mvcc.cc:205] Tried to move back new op lower bound from 7022801334523289600 to 7022801332238249984. Current Snapshot: MvccSnapshot[applied={T|T < 7022801334523289600}]
INSERT report
rows total: 2000
time total: 217.55 ms
time per row: 0.108775 ms
-----------------------------------------------------
Suppressions used:
count bytes template
14 3624 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:48.209298 12829 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:48.218714 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 31b3831e0f36425982b1627dd5508c99 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.218932 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 31b3831e0f36425982b1627dd5508c99 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.219228 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b3831e0f36425982b1627dd5508c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.219331 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b3831e0f36425982b1627dd5508c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.219385 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 74d6cce81d164722afad1935d4695374 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.219717 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d6cce81d164722afad1935d4695374. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.223835 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 31b3831e0f36425982b1627dd5508c99 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.223838 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 74d6cce81d164722afad1935d4695374 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.224246 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31b3831e0f36425982b1627dd5508c99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.224246 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d6cce81d164722afad1935d4695374. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.224678 13083 tablet_service.cc:1467] Processing CreateTablet for tablet 74d6cce81d164722afad1935d4695374 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.225032 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d6cce81d164722afad1935d4695374. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.227397 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 7a98388bf2fd402b8064d2cca06f5b8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.227891 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a98388bf2fd402b8064d2cca06f5b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.229699 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 7a98388bf2fd402b8064d2cca06f5b8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.230119 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a98388bf2fd402b8064d2cca06f5b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.230886 14238 tablet_bootstrap.cc:492] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:48.228873 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 7a98388bf2fd402b8064d2cca06f5b8b (DEFAULT_TABLE table=rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:48.233309 14238 tablet_bootstrap.cc:654] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.233640 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a98388bf2fd402b8064d2cca06f5b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:48.239926 14238 tablet_bootstrap.cc:492] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:48.240216 14238 ts_tablet_manager.cc:1367] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240501 08:11:48.241050 14238 raft_consensus.cc:348] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:48.241395 14238 raft_consensus.cc:374] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.241518 14238 raft_consensus.cc:729] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.241864 14238 consensus_queue.cc:260] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:48.242635 14240 tablet_bootstrap.cc:492] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:48.243651 14238 ts_tablet_manager.cc:1398] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:48.243943 14238 tablet_bootstrap.cc:492] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:48.244473 14240 tablet_bootstrap.cc:654] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.245730 14238 tablet_bootstrap.cc:654] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.246502 14242 tablet_bootstrap.cc:492] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:48.251422 14239 tablet_bootstrap.cc:492] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:48.251581 14242 tablet_bootstrap.cc:654] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.253862 14240 tablet_bootstrap.cc:492] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:48.254050 14239 tablet_bootstrap.cc:654] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.254149 14240 ts_tablet_manager.cc:1367] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240501 08:11:48.255276 14240 raft_consensus.cc:348] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:48.255503 14240 raft_consensus.cc:374] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.255640 14240 raft_consensus.cc:729] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.255952 14240 consensus_queue.cc:260] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:48.256588 14238 tablet_bootstrap.cc:492] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:48.256600 14240 ts_tablet_manager.cc:1398] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:48.256772 14238 ts_tablet_manager.cc:1367] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240501 08:11:48.257094 14240 tablet_bootstrap.cc:492] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:48.257655 14238 raft_consensus.cc:348] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.257887 14238 raft_consensus.cc:374] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.257980 14238 raft_consensus.cc:729] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.258631 14239 tablet_bootstrap.cc:492] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:48.258864 14239 ts_tablet_manager.cc:1367] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:11:48.258970 14240 tablet_bootstrap.cc:654] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.259716 14239 raft_consensus.cc:348] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:48.260001 14239 raft_consensus.cc:374] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.260169 14239 raft_consensus.cc:729] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.260067 14238 consensus_queue.cc:260] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.260564 14239 consensus_queue.cc:260] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:48.262434 14239 ts_tablet_manager.cc:1398] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:48.263721 14239 tablet_bootstrap.cc:492] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:48.263480 14242 tablet_bootstrap.cc:492] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:48.264680 14242 ts_tablet_manager.cc:1367] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240501 08:11:48.264730 14238 ts_tablet_manager.cc:1398] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:11:48.265622 14242 raft_consensus.cc:348] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.265859 14239 tablet_bootstrap.cc:654] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.265930 14242 raft_consensus.cc:374] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.266070 14242 raft_consensus.cc:729] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.266417 14242 consensus_queue.cc:260] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.268103 14238 tablet_bootstrap.cc:492] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:48.270345 14238 tablet_bootstrap.cc:654] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.272119 14242 ts_tablet_manager.cc:1398] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:11:48.273260 14242 tablet_bootstrap.cc:492] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:48.273959 14238 tablet_bootstrap.cc:492] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:48.274114 14238 ts_tablet_manager.cc:1367] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:11:48.274816 14242 tablet_bootstrap.cc:654] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:48.274940 14238 raft_consensus.cc:348] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.275168 14238 raft_consensus.cc:374] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.275259 14238 raft_consensus.cc:729] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.275619 14238 consensus_queue.cc:260] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.276829 14242 tablet_bootstrap.cc:492] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:48.276899 14240 tablet_bootstrap.cc:492] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:48.276944 14238 ts_tablet_manager.cc:1398] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:48.276994 14242 ts_tablet_manager.cc:1367] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:48.277074 14240 ts_tablet_manager.cc:1367] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.003s
I20240501 08:11:48.277627 14242 raft_consensus.cc:348] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.277808 14242 raft_consensus.cc:374] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.277841 14239 tablet_bootstrap.cc:492] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:48.277860 14242 raft_consensus.cc:729] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.277823 14240 raft_consensus.cc:348] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.278036 14239 ts_tablet_manager.cc:1367] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240501 08:11:48.278039 14240 raft_consensus.cc:374] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.278040 14242 consensus_queue.cc:260] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.278108 14240 raft_consensus.cc:729] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.278401 14240 consensus_queue.cc:260] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.278548 14242 ts_tablet_manager.cc:1398] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:48.278856 14239 raft_consensus.cc:348] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.279006 14240 ts_tablet_manager.cc:1398] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:48.279132 14239 raft_consensus.cc:374] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:48.279243 14239 raft_consensus.cc:729] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:48.279506 14239 consensus_queue.cc:260] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.280076 14239 ts_tablet_manager.cc:1398] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:48.333803 14248 raft_consensus.cc:482] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:48.334007 14248 raft_consensus.cc:504] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:48.335021 14248 leader_election.cc:288] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:48.335716 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b3831e0f36425982b1627dd5508c99" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:48.336025 13105 raft_consensus.cc:2454] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:48.336531 12909 leader_election.cc:302] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:48.336844 14248 raft_consensus.cc:2781] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:48.336975 14248 raft_consensus.cc:482] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:48.337098 14248 raft_consensus.cc:3037] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.339123 14248 raft_consensus.cc:504] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:48.339733 14248 leader_election.cc:288] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:48.340257 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b3831e0f36425982b1627dd5508c99" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:48.340529 13105 raft_consensus.cc:3037] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.342082 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b3831e0f36425982b1627dd5508c99" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:48.342180 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b3831e0f36425982b1627dd5508c99" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:48.342370 13234 raft_consensus.cc:2454] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:48.343024 13105 raft_consensus.cc:2454] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:48.343372 12909 leader_election.cc:302] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters: 6ed448d04e4646658c0de37d829ff15b
I20240501 08:11:48.343641 14248 raft_consensus.cc:2781] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:48.344070 14248 raft_consensus.cc:686] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:11:48.344584 14248 consensus_queue.cc:237] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:48.347710 12830 catalog_manager.cc:5484] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:11:48.469597 14135 raft_consensus.cc:482] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:48.469795 14135 raft_consensus.cc:504] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.470467 14135 leader_election.cc:288] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:48.471103 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a98388bf2fd402b8064d2cca06f5b8b" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:48.471098 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a98388bf2fd402b8064d2cca06f5b8b" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:48.471354 12973 raft_consensus.cc:2454] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:48.471350 13234 raft_consensus.cc:2454] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:48.471776 13301 leader_election.cc:302] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:48.472075 14135 raft_consensus.cc:2781] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:48.472198 14135 raft_consensus.cc:482] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:48.472321 14135 raft_consensus.cc:3037] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.474398 14135 raft_consensus.cc:504] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.475050 14135 leader_election.cc:288] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:48.475503 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a98388bf2fd402b8064d2cca06f5b8b" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:48.475731 12973 raft_consensus.cc:3037] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.475752 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a98388bf2fd402b8064d2cca06f5b8b" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:48.475987 13234 raft_consensus.cc:3037] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.477752 12973 raft_consensus.cc:2454] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:48.478204 13301 leader_election.cc:302] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:48.478510 14135 raft_consensus.cc:2781] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:48.478575 13234 raft_consensus.cc:2454] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:48.479025 14135 raft_consensus.cc:686] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:11:48.479746 14135 consensus_queue.cc:237] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.482445 12830 catalog_manager.cc:5484] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:11:48.539968 14135 raft_consensus.cc:482] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:48.540220 14135 raft_consensus.cc:504] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.540918 14135 leader_election.cc:288] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:48.541472 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d6cce81d164722afad1935d4695374" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:48.541541 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d6cce81d164722afad1935d4695374" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:48.541672 13105 raft_consensus.cc:2454] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:48.541811 13234 raft_consensus.cc:2454] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:48.542105 13300 leader_election.cc:302] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:48.542426 14135 raft_consensus.cc:2781] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:48.542606 14135 raft_consensus.cc:482] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:48.542726 14135 raft_consensus.cc:3037] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.544968 14135 raft_consensus.cc:504] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.545691 14135 leader_election.cc:288] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:48.546070 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d6cce81d164722afad1935d4695374" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:48.546308 13105 raft_consensus.cc:3037] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.546411 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d6cce81d164722afad1935d4695374" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:48.546650 13234 raft_consensus.cc:3037] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.548360 13105 raft_consensus.cc:2454] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:48.548537 13234 raft_consensus.cc:2454] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:48.548810 13300 leader_election.cc:302] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:48.549160 14135 raft_consensus.cc:2781] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:48.549363 14135 raft_consensus.cc:686] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:11:48.549597 14135 consensus_queue.cc:237] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:48.552351 12830 catalog_manager.cc:5484] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:11:48.883574 14256 consensus_queue.cc:1035] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:48.890398 14256 consensus_queue.cc:1035] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:48.908427 13234 raft_consensus.cc:3037] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:48.912575 14264 consensus_queue.cc:1035] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:48.915035 14264 consensus_queue.cc:1035] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:48.972796 14260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:48.973191 14260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:48.980940 14260 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:48.990556 14135 consensus_queue.cc:1035] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:48.995958 14256 consensus_queue.cc:1035] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 245.417 ms
time per row: 0.122709 ms
I20240501 08:11:49.382396 12824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:49.392122 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 67d42d76dff948b98a373831b515c7f7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.392379 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 67d42d76dff948b98a373831b515c7f7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.392621 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d42d76dff948b98a373831b515c7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.392625 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 67d42d76dff948b98a373831b515c7f7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.392793 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d42d76dff948b98a373831b515c7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.393082 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d42d76dff948b98a373831b515c7f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.393630 13213 tablet_service.cc:1467] Processing CreateTablet for tablet ca8689eeb65744db8404fffcc84517b0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.394003 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8689eeb65744db8404fffcc84517b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.397563 13083 tablet_service.cc:1467] Processing CreateTablet for tablet ecda2d4abaac4ed6a31c5acff14177f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.398016 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecda2d4abaac4ed6a31c5acff14177f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.399760 13345 tablet_service.cc:1467] Processing CreateTablet for tablet ecda2d4abaac4ed6a31c5acff14177f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.400211 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecda2d4abaac4ed6a31c5acff14177f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.401934 13085 tablet_service.cc:1467] Processing CreateTablet for tablet ca8689eeb65744db8404fffcc84517b0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.402374 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8689eeb65744db8404fffcc84517b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.402333 13344 tablet_service.cc:1467] Processing CreateTablet for tablet ca8689eeb65744db8404fffcc84517b0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.402838 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca8689eeb65744db8404fffcc84517b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.405005 12954 tablet_service.cc:1467] Processing CreateTablet for tablet ecda2d4abaac4ed6a31c5acff14177f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:49.405454 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecda2d4abaac4ed6a31c5acff14177f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:49.406728 14324 tablet_bootstrap.cc:492] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:49.408602 14324 tablet_bootstrap.cc:654] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.415145 14325 tablet_bootstrap.cc:492] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:49.417713 14325 tablet_bootstrap.cc:654] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.428817 14325 tablet_bootstrap.cc:492] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:49.429132 14325 ts_tablet_manager.cc:1367] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240501 08:11:49.430215 14325 raft_consensus.cc:348] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:49.435135 14323 tablet_bootstrap.cc:492] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:49.435699 14325 raft_consensus.cc:374] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.435854 14325 raft_consensus.cc:729] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.436153 14325 consensus_queue.cc:260] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:49.436846 14325 ts_tablet_manager.cc:1398] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:11:49.437681 14325 tablet_bootstrap.cc:492] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:49.439304 14325 tablet_bootstrap.cc:654] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.447691 14325 tablet_bootstrap.cc:492] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:49.448004 14325 ts_tablet_manager.cc:1367] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240501 08:11:49.448876 14325 raft_consensus.cc:348] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:49.449219 14325 raft_consensus.cc:374] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.449344 14325 raft_consensus.cc:729] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.449512 14327 tablet_bootstrap.cc:492] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:49.449715 14325 consensus_queue.cc:260] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:49.450451 14325 ts_tablet_manager.cc:1398] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:49.451593 14327 tablet_bootstrap.cc:654] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.453585 14323 tablet_bootstrap.cc:654] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.460117 14324 tablet_bootstrap.cc:492] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:49.460389 14324 ts_tablet_manager.cc:1367] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.054s user 0.025s sys 0.016s
I20240501 08:11:49.461336 14324 raft_consensus.cc:348] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:49.461584 14324 raft_consensus.cc:374] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.461687 14324 raft_consensus.cc:729] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.461732 14327 tablet_bootstrap.cc:492] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:49.461877 14327 ts_tablet_manager.cc:1367] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240501 08:11:49.461987 14324 consensus_queue.cc:260] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:49.462819 14327 raft_consensus.cc:348] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:49.463290 14327 raft_consensus.cc:374] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.463398 14327 raft_consensus.cc:729] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.463703 14327 consensus_queue.cc:260] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:49.464257 14327 ts_tablet_manager.cc:1398] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:49.464628 14327 tablet_bootstrap.cc:492] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:49.466678 14327 tablet_bootstrap.cc:654] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.465412 14324 ts_tablet_manager.cc:1398] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:11:49.469321 14324 tablet_bootstrap.cc:492] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:49.469532 14323 tablet_bootstrap.cc:492] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:49.469797 14323 ts_tablet_manager.cc:1367] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.035s user 0.012s sys 0.000s
I20240501 08:11:49.470821 14323 raft_consensus.cc:348] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:49.471155 14324 tablet_bootstrap.cc:654] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.471141 14323 raft_consensus.cc:374] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.471266 14323 raft_consensus.cc:729] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.471518 14323 consensus_queue.cc:260] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:49.472203 14323 ts_tablet_manager.cc:1398] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:49.472535 14323 tablet_bootstrap.cc:492] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:49.474272 14323 tablet_bootstrap.cc:654] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.476264 14327 tablet_bootstrap.cc:492] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:49.476469 14327 ts_tablet_manager.cc:1367] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240501 08:11:49.477416 14327 raft_consensus.cc:348] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:49.477649 14327 raft_consensus.cc:374] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.477787 14327 raft_consensus.cc:729] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.478082 14327 consensus_queue.cc:260] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:49.478807 14327 ts_tablet_manager.cc:1398] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:49.479118 14327 tablet_bootstrap.cc:492] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:49.479347 14323 tablet_bootstrap.cc:492] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:49.479507 14323 ts_tablet_manager.cc:1367] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:11:49.480222 14323 raft_consensus.cc:348] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:49.480492 14323 raft_consensus.cc:374] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.480576 14323 raft_consensus.cc:729] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.480727 14327 tablet_bootstrap.cc:654] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:49.480859 14323 consensus_queue.cc:260] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:49.478960 14324 tablet_bootstrap.cc:492] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:49.481389 14324 ts_tablet_manager.cc:1367] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.001s
I20240501 08:11:49.482080 14323 ts_tablet_manager.cc:1398] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:49.483255 14324 raft_consensus.cc:348] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:49.483485 14324 raft_consensus.cc:374] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.483556 14324 raft_consensus.cc:729] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.483794 14324 consensus_queue.cc:260] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:49.484375 14324 ts_tablet_manager.cc:1398] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:49.487409 14327 tablet_bootstrap.cc:492] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:49.487589 14327 ts_tablet_manager.cc:1367] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:11:49.488516 14327 raft_consensus.cc:348] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:49.488739 14327 raft_consensus.cc:374] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:49.488843 14327 raft_consensus.cc:729] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:49.489114 14327 consensus_queue.cc:260] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:49.489717 14327 ts_tablet_manager.cc:1398] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:49.495383 14223 raft_consensus.cc:482] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:49.495537 14223 raft_consensus.cc:504] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:49.496129 14223 leader_election.cc:288] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:49.496649 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8689eeb65744db8404fffcc84517b0" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:49.496666 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8689eeb65744db8404fffcc84517b0" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:49.496841 13364 raft_consensus.cc:2454] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:49.496908 13235 raft_consensus.cc:2454] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:49.497251 13040 leader_election.cc:302] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:49.497524 14223 raft_consensus.cc:2781] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:49.497620 14223 raft_consensus.cc:482] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:49.497711 14223 raft_consensus.cc:3037] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.499624 14223 raft_consensus.cc:504] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:49.500226 14223 leader_election.cc:288] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:49.500803 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8689eeb65744db8404fffcc84517b0" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:49.501008 13364 raft_consensus.cc:3037] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.503075 13364 raft_consensus.cc:2454] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:49.503468 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca8689eeb65744db8404fffcc84517b0" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:49.503525 13040 leader_election.cc:302] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:49.503695 13235 raft_consensus.cc:3037] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.503839 14223 raft_consensus.cc:2781] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:49.504359 14223 raft_consensus.cc:686] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:49.505105 14223 consensus_queue.cc:237] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:49.506655 13235 raft_consensus.cc:2454] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:49.508471 12832 catalog_manager.cc:5484] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:11:49.616582 14223 raft_consensus.cc:482] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:49.616798 14223 raft_consensus.cc:504] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:49.617548 14223 leader_election.cc:288] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:49.618052 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d42d76dff948b98a373831b515c7f7" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:49.618157 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d42d76dff948b98a373831b515c7f7" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:49.618290 12973 raft_consensus.cc:2454] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:49.618409 13235 raft_consensus.cc:2454] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:49.618741 13041 leader_election.cc:302] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:49.619009 14223 raft_consensus.cc:2781] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:49.619169 14223 raft_consensus.cc:482] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:49.619274 14223 raft_consensus.cc:3037] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.621291 14223 raft_consensus.cc:504] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:49.621853 14223 leader_election.cc:288] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:49.622351 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d42d76dff948b98a373831b515c7f7" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:49.622351 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67d42d76dff948b98a373831b515c7f7" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:49.622548 13235 raft_consensus.cc:3037] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.622567 12973 raft_consensus.cc:3037] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.625481 12973 raft_consensus.cc:2454] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:49.625499 13235 raft_consensus.cc:2454] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:49.625945 13041 leader_election.cc:302] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:49.626232 14223 raft_consensus.cc:2781] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:49.626399 14223 raft_consensus.cc:686] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:49.626660 14223 consensus_queue.cc:237] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:49.629613 12832 catalog_manager.cc:5484] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:49.665215 14223 raft_consensus.cc:482] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:49.665551 14223 raft_consensus.cc:504] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:49.666240 14223 leader_election.cc:288] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:49.666783 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecda2d4abaac4ed6a31c5acff14177f3" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:49.666797 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecda2d4abaac4ed6a31c5acff14177f3" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:49.667016 12973 raft_consensus.cc:2454] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:49.667048 13364 raft_consensus.cc:2454] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:49.667469 13041 leader_election.cc:302] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:49.667763 14223 raft_consensus.cc:2781] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:49.667879 14223 raft_consensus.cc:482] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:49.667953 14223 raft_consensus.cc:3037] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.670001 14223 raft_consensus.cc:504] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:49.670553 14223 leader_election.cc:288] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:49.671059 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecda2d4abaac4ed6a31c5acff14177f3" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:49.671083 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecda2d4abaac4ed6a31c5acff14177f3" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:49.671253 12973 raft_consensus.cc:3037] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.671263 13364 raft_consensus.cc:3037] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:49.673173 12973 raft_consensus.cc:2454] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:49.673243 13364 raft_consensus.cc:2454] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:49.673581 13041 leader_election.cc:302] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:49.673868 14223 raft_consensus.cc:2781] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:49.674006 14223 raft_consensus.cc:686] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:49.674291 14223 consensus_queue.cc:237] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:49.677289 12832 catalog_manager.cc:5484] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:11:49.980458 14223 consensus_queue.cc:1035] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:49.987150 14223 consensus_queue.cc:1035] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:50.081539 14340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:50.081858 14340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:50.089285 14340 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:50.116912 14223 consensus_queue.cc:1035] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240501 08:11:50.123178 14223 consensus_queue.cc:1035] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:50.160432 14223 consensus_queue.cc:1035] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:50.171279 14336 consensus_queue.cc:1035] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 215.829 ms
time per row: 0.107915 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:51.417155 12832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:51.426503 13213 tablet_service.cc:1467] Processing CreateTablet for tablet cd9a5b5333d54f6d9d8186e4d9b1b1be (DEFAULT_TABLE table=rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.426486 12953 tablet_service.cc:1467] Processing CreateTablet for tablet cd9a5b5333d54f6d9d8186e4d9b1b1be (DEFAULT_TABLE table=rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.426972 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9a5b5333d54f6d9d8186e4d9b1b1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.426990 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9a5b5333d54f6d9d8186e4d9b1b1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.428010 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 94e7fb526fbc4ebc94875c00b49ac43d (DEFAULT_TABLE table=rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.428153 13344 tablet_service.cc:1467] Processing CreateTablet for tablet cd9a5b5333d54f6d9d8186e4d9b1b1be (DEFAULT_TABLE table=rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.428468 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e7fb526fbc4ebc94875c00b49ac43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.428588 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9a5b5333d54f6d9d8186e4d9b1b1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.428683 13345 tablet_service.cc:1467] Processing CreateTablet for tablet e6fefd76544642ddb787cc6febe8df82 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.429054 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fefd76544642ddb787cc6febe8df82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.433678 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 94e7fb526fbc4ebc94875c00b49ac43d (DEFAULT_TABLE table=rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.434109 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e7fb526fbc4ebc94875c00b49ac43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.436522 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 94e7fb526fbc4ebc94875c00b49ac43d (DEFAULT_TABLE table=rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.436591 12954 tablet_service.cc:1467] Processing CreateTablet for tablet e6fefd76544642ddb787cc6febe8df82 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.436996 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fefd76544642ddb787cc6febe8df82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.437088 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e7fb526fbc4ebc94875c00b49ac43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.438155 14408 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:51.439795 14408 tablet_bootstrap.cc:654] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.436007 13215 tablet_service.cc:1467] Processing CreateTablet for tablet e6fefd76544642ddb787cc6febe8df82 (DEFAULT_TABLE table=rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:51.446853 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fefd76544642ddb787cc6febe8df82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:51.472232 14411 tablet_bootstrap.cc:492] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:51.477869 14410 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:51.479720 14411 tablet_bootstrap.cc:654] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.480576 14410 tablet_bootstrap.cc:654] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.481664 14407 tablet_bootstrap.cc:492] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:51.484015 14407 tablet_bootstrap.cc:654] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.486635 14408 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:51.486884 14408 ts_tablet_manager.cc:1367] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.049s user 0.019s sys 0.027s
I20240501 08:11:51.487830 14408 raft_consensus.cc:348] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:51.488219 14408 raft_consensus.cc:374] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.488426 14408 raft_consensus.cc:729] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.488821 14408 consensus_queue.cc:260] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:51.489696 14408 ts_tablet_manager.cc:1398] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:51.489955 14410 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:51.489964 14407 tablet_bootstrap.cc:492] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:51.490115 14407 ts_tablet_manager.cc:1367] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240501 08:11:51.490195 14410 ts_tablet_manager.cc:1367] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240501 08:11:51.490934 14410 raft_consensus.cc:348] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:51.491109 14410 raft_consensus.cc:374] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.491154 14411 tablet_bootstrap.cc:492] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:51.491257 14410 raft_consensus.cc:729] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.491384 14411 ts_tablet_manager.cc:1367] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.007s
I20240501 08:11:51.491575 14410 consensus_queue.cc:260] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:51.491721 14407 raft_consensus.cc:348] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:51.491940 14407 raft_consensus.cc:374] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.490034 14408 tablet_bootstrap.cc:492] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:51.492267 14411 raft_consensus.cc:348] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:51.492514 14411 raft_consensus.cc:374] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.492633 14411 raft_consensus.cc:729] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.492758 14407 raft_consensus.cc:729] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.492892 14411 consensus_queue.cc:260] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:51.493083 14407 consensus_queue.cc:260] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:51.493294 14410 ts_tablet_manager.cc:1398] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:11:51.493610 14410 tablet_bootstrap.cc:492] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:51.493695 14407 ts_tablet_manager.cc:1398] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:11:51.493736 14411 ts_tablet_manager.cc:1398] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:51.494057 14411 tablet_bootstrap.cc:492] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:51.495824 14411 tablet_bootstrap.cc:654] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.496043 14410 tablet_bootstrap.cc:654] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.499261 14408 tablet_bootstrap.cc:654] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.507927 14408 tablet_bootstrap.cc:492] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:51.508282 14408 ts_tablet_manager.cc:1367] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.000s
I20240501 08:11:51.509409 14408 raft_consensus.cc:348] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:51.509760 14408 raft_consensus.cc:374] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.509889 14408 raft_consensus.cc:729] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.510243 14408 consensus_queue.cc:260] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:51.512151 14408 ts_tablet_manager.cc:1398] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:51.516356 14411 tablet_bootstrap.cc:492] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:51.516602 14411 ts_tablet_manager.cc:1367] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.009s
I20240501 08:11:51.517017 14410 tablet_bootstrap.cc:492] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:51.517208 14410 ts_tablet_manager.cc:1367] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.017s
I20240501 08:11:51.517480 14411 raft_consensus.cc:348] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:51.517706 14411 raft_consensus.cc:374] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.517791 14411 raft_consensus.cc:729] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.518061 14410 raft_consensus.cc:348] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:51.518043 14411 consensus_queue.cc:260] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:51.518277 14410 raft_consensus.cc:374] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.518363 14410 raft_consensus.cc:729] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.518676 14411 ts_tablet_manager.cc:1398] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:51.518606 14410 consensus_queue.cc:260] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:51.518991 14411 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:51.519208 14410 ts_tablet_manager.cc:1398] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:51.519634 14410 tablet_bootstrap.cc:492] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:51.520610 14411 tablet_bootstrap.cc:654] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.521281 14410 tablet_bootstrap.cc:654] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:51.527699 14410 tablet_bootstrap.cc:492] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:51.527871 14410 ts_tablet_manager.cc:1367] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 08:11:51.528589 14411 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:51.528760 14411 ts_tablet_manager.cc:1367] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240501 08:11:51.528642 14410 raft_consensus.cc:348] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:51.528848 14410 raft_consensus.cc:374] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.528946 14410 raft_consensus.cc:729] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.529186 14410 consensus_queue.cc:260] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:51.529603 14411 raft_consensus.cc:348] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:51.529768 14410 ts_tablet_manager.cc:1398] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:51.529783 14411 raft_consensus.cc:374] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:51.529867 14411 raft_consensus.cc:729] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:51.530093 14411 consensus_queue.cc:260] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:51.530634 14411 ts_tablet_manager.cc:1398] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:11:51.540786 14169 raft_consensus.cc:482] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:51.540973 14169 raft_consensus.cc:504] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:51.541626 14169 leader_election.cc:288] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:51.542138 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e7fb526fbc4ebc94875c00b49ac43d" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:51.542152 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e7fb526fbc4ebc94875c00b49ac43d" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:51.542367 12973 raft_consensus.cc:2454] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:51.542382 13105 raft_consensus.cc:2454] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:51.542850 13171 leader_election.cc:302] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:51.543113 14169 raft_consensus.cc:2781] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:51.543221 14169 raft_consensus.cc:482] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:51.543309 14169 raft_consensus.cc:3037] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.545203 14169 raft_consensus.cc:504] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:51.545733 14169 leader_election.cc:288] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:51.546217 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e7fb526fbc4ebc94875c00b49ac43d" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:51.546217 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e7fb526fbc4ebc94875c00b49ac43d" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:51.546445 13105 raft_consensus.cc:3037] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.546445 12973 raft_consensus.cc:3037] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.549281 13105 raft_consensus.cc:2454] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:51.549273 12973 raft_consensus.cc:2454] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:51.549624 13171 leader_election.cc:302] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:51.549916 14169 raft_consensus.cc:2781] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:51.550302 14169 raft_consensus.cc:686] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:11:51.550591 14169 consensus_queue.cc:237] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:51.551316 14420 raft_consensus.cc:482] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:51.551427 14420 raft_consensus.cc:504] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:51.552201 14420 leader_election.cc:288] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:51.552769 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fefd76544642ddb787cc6febe8df82" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:51.553005 13364 raft_consensus.cc:2454] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:51.553433 13170 leader_election.cc:302] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:11:51.553692 14420 raft_consensus.cc:2781] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:51.553843 14420 raft_consensus.cc:482] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:51.553946 14420 raft_consensus.cc:3037] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.553879 12828 catalog_manager.cc:5484] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:11:51.555298 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fefd76544642ddb787cc6febe8df82" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:51.555850 12973 raft_consensus.cc:2454] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:51.578241 14420 raft_consensus.cc:504] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:51.579066 14420 leader_election.cc:288] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:51.579618 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fefd76544642ddb787cc6febe8df82" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:51.579615 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fefd76544642ddb787cc6febe8df82" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:51.579838 13364 raft_consensus.cc:3037] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.579870 12973 raft_consensus.cc:3037] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.581867 12973 raft_consensus.cc:2454] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:51.581868 13364 raft_consensus.cc:2454] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:51.582340 13171 leader_election.cc:302] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:51.582710 14420 raft_consensus.cc:2781] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:51.582856 14420 raft_consensus.cc:686] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:11:51.583155 14420 consensus_queue.cc:237] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:51.585978 12828 catalog_manager.cc:5484] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:51.741791 14269 raft_consensus.cc:482] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:51.742002 14269 raft_consensus.cc:504] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:51.742786 14269 leader_election.cc:288] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:51.743409 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9a5b5333d54f6d9d8186e4d9b1b1be" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:51.743386 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9a5b5333d54f6d9d8186e4d9b1b1be" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:51.743657 13364 raft_consensus.cc:2454] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:51.743654 13235 raft_consensus.cc:2454] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:51.744076 12909 leader_election.cc:302] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:51.744443 14269 raft_consensus.cc:2781] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:51.744546 14269 raft_consensus.cc:482] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:51.744645 14269 raft_consensus.cc:3037] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.746547 14269 raft_consensus.cc:504] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:51.747068 14269 leader_election.cc:288] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:51.747484 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9a5b5333d54f6d9d8186e4d9b1b1be" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:51.747563 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9a5b5333d54f6d9d8186e4d9b1b1be" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:51.747696 13235 raft_consensus.cc:3037] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.747785 13364 raft_consensus.cc:3037] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:51.749701 13235 raft_consensus.cc:2454] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:51.749758 13364 raft_consensus.cc:2454] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:51.750092 12909 leader_election.cc:302] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:51.750435 14269 raft_consensus.cc:2781] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:51.750816 14269 raft_consensus.cc:686] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:11:51.751099 14269 consensus_queue.cc:237] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:51.753732 12828 catalog_manager.cc:5484] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:11:51.971374 14169 consensus_queue.cc:1035] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:51.976027 14169 consensus_queue.cc:1035] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:52.147481 14169 consensus_queue.cc:1035] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:52.152201 14169 consensus_queue.cc:1035] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:52.162309 14426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:52.162729 14426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:52.170262 14426 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 08:11:52.289500 14269 consensus_queue.cc:1035] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:52.305239 14462 consensus_queue.cc:1035] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:52.391135 13301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 60924.1us
INSERT report
rows total: 2000
time total: 313.162 ms
time per row: 0.156581 ms
I20240501 08:11:52.637428 12824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:52.648797 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 65842be1fe3c469dba1b57afdb5fb948 (DEFAULT_TABLE table=rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:52.649046 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 65842be1fe3c469dba1b57afdb5fb948 (DEFAULT_TABLE table=rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:52.649304 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65842be1fe3c469dba1b57afdb5fb948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:52.649482 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65842be1fe3c469dba1b57afdb5fb948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:52.649780 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 75a875e42dcb4b6e851dd7ab8589a03b (DEFAULT_TABLE table=rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:52.650187 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a875e42dcb4b6e851dd7ab8589a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:52.650285 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 22dc5fac8a454c85af08f20c30ff96ff (DEFAULT_TABLE table=rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:52.650701 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22dc5fac8a454c85af08f20c30ff96ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:52.653599 13083 tablet_service.cc:1467] Processing CreateTablet for tablet 75a875e42dcb4b6e851dd7ab8589a03b (DEFAULT_TABLE table=rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:52.653817 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 65842be1fe3c469dba1b57afdb5fb948 (DEFAULT_TABLE table=rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:52.654013 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a875e42dcb4b6e851dd7ab8589a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:52.654212 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65842be1fe3c469dba1b57afdb5fb948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:52.656644 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 22dc5fac8a454c85af08f20c30ff96ff (DEFAULT_TABLE table=rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:52.657074 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22dc5fac8a454c85af08f20c30ff96ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:52.660863 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 75a875e42dcb4b6e851dd7ab8589a03b (DEFAULT_TABLE table=rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:52.661367 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a875e42dcb4b6e851dd7ab8589a03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:52.667446 14495 tablet_bootstrap.cc:492] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:52.667512 14496 tablet_bootstrap.cc:492] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:52.668370 14494 tablet_bootstrap.cc:492] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:52.670207 14494 tablet_bootstrap.cc:654] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:52.670336 14495 tablet_bootstrap.cc:654] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:52.670526 14496 tablet_bootstrap.cc:654] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:52.681103 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 22dc5fac8a454c85af08f20c30ff96ff (DEFAULT_TABLE table=rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:52.681674 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22dc5fac8a454c85af08f20c30ff96ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:52.681752 14496 tablet_bootstrap.cc:492] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:52.681890 14496 ts_tablet_manager.cc:1367] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240501 08:11:52.682792 14496 raft_consensus.cc:348] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.683005 14496 raft_consensus.cc:374] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:52.683100 14496 raft_consensus.cc:729] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:52.683352 14496 consensus_queue.cc:260] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.686730 14496 ts_tablet_manager.cc:1398] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:52.686767 14493 tablet_bootstrap.cc:492] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:52.687093 14496 tablet_bootstrap.cc:492] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:52.688773 14493 tablet_bootstrap.cc:654] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:52.688854 14496 tablet_bootstrap.cc:654] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:52.691218 14495 tablet_bootstrap.cc:492] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:52.691433 14495 ts_tablet_manager.cc:1367] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20240501 08:11:52.692245 14495 raft_consensus.cc:348] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:52.692549 14494 tablet_bootstrap.cc:492] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:52.692549 14495 raft_consensus.cc:374] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:52.692651 14495 raft_consensus.cc:729] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:52.692762 14494 ts_tablet_manager.cc:1367] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.008s
I20240501 08:11:52.692970 14495 consensus_queue.cc:260] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:52.693737 14495 ts_tablet_manager.cc:1398] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:52.693657 14494 raft_consensus.cc:348] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.693905 14494 raft_consensus.cc:374] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:52.694031 14494 raft_consensus.cc:729] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:52.694278 14494 consensus_queue.cc:260] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.694867 14494 ts_tablet_manager.cc:1398] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:52.695780 14494 tablet_bootstrap.cc:492] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:52.698544 14496 tablet_bootstrap.cc:492] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:52.698746 14496 ts_tablet_manager.cc:1367] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240501 08:11:52.699478 14494 tablet_bootstrap.cc:654] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:52.699687 14496 raft_consensus.cc:348] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:52.699916 14496 raft_consensus.cc:374] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:52.699993 14496 raft_consensus.cc:729] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:52.700309 14496 consensus_queue.cc:260] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:52.700760 14493 tablet_bootstrap.cc:492] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:52.700963 14493 ts_tablet_manager.cc:1367] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20240501 08:11:52.701108 14496 ts_tablet_manager.cc:1398] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 08:11:52.701575 14496 tablet_bootstrap.cc:492] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:52.701903 14493 raft_consensus.cc:348] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.702172 14493 raft_consensus.cc:374] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:52.702304 14493 raft_consensus.cc:729] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:52.702638 14493 consensus_queue.cc:260] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.703348 14493 ts_tablet_manager.cc:1398] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:52.703682 14493 tablet_bootstrap.cc:492] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:52.704880 14496 tablet_bootstrap.cc:654] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:52.705338 14493 tablet_bootstrap.cc:654] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:52.710706 14496 tablet_bootstrap.cc:492] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:52.710898 14496 ts_tablet_manager.cc:1367] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240501 08:11:52.711961 14496 raft_consensus.cc:348] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.712240 14496 raft_consensus.cc:374] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:52.712388 14496 raft_consensus.cc:729] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:52.712674 14496 consensus_queue.cc:260] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.713527 14496 ts_tablet_manager.cc:1398] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:52.715121 14493 tablet_bootstrap.cc:492] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:52.715273 14494 tablet_bootstrap.cc:492] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:52.715304 14493 ts_tablet_manager.cc:1367] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240501 08:11:52.715530 14494 ts_tablet_manager.cc:1367] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.007s
I20240501 08:11:52.716089 14493 raft_consensus.cc:348] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.716346 14493 raft_consensus.cc:374] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:52.716425 14493 raft_consensus.cc:729] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:52.716337 14494 raft_consensus.cc:348] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:52.716725 14494 raft_consensus.cc:374] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:52.716686 14493 consensus_queue.cc:260] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.716833 14494 raft_consensus.cc:729] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:52.717177 14494 consensus_queue.cc:260] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:52.717964 14493 ts_tablet_manager.cc:1398] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:11:52.718111 14494 ts_tablet_manager.cc:1398] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:52.718432 14494 tablet_bootstrap.cc:492] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:52.719902 14494 tablet_bootstrap.cc:654] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:52.723901 14462 raft_consensus.cc:482] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:52.724064 14462 raft_consensus.cc:504] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.724388 14494 tablet_bootstrap.cc:492] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:52.724529 14494 ts_tablet_manager.cc:1367] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20240501 08:11:52.724803 14462 leader_election.cc:288] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:52.725195 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22dc5fac8a454c85af08f20c30ff96ff" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:52.725286 14494 raft_consensus.cc:348] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.725346 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22dc5fac8a454c85af08f20c30ff96ff" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:52.725499 14494 raft_consensus.cc:374] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:52.725557 13364 raft_consensus.cc:2454] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:52.725598 14494 raft_consensus.cc:729] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:52.725811 14494 consensus_queue.cc:260] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.725958 12909 leader_election.cc:302] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:52.726063 13105 raft_consensus.cc:2454] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:52.726315 14462 raft_consensus.cc:2781] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:52.726415 14462 raft_consensus.cc:482] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:52.726509 14462 raft_consensus.cc:3037] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:52.726769 14494 ts_tablet_manager.cc:1398] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:52.729176 14462 raft_consensus.cc:504] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.729750 14462 leader_election.cc:288] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:52.730379 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22dc5fac8a454c85af08f20c30ff96ff" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:52.730504 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22dc5fac8a454c85af08f20c30ff96ff" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:52.730580 13105 raft_consensus.cc:3037] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:52.730707 13364 raft_consensus.cc:3037] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:52.733444 13105 raft_consensus.cc:2454] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:52.733814 12909 leader_election.cc:302] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:52.733961 13364 raft_consensus.cc:2454] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:52.734138 14462 raft_consensus.cc:2781] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:52.734323 14462 raft_consensus.cc:686] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:11:52.734627 14462 consensus_queue.cc:237] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.738029 12824 catalog_manager.cc:5484] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:11:52.773293 14480 raft_consensus.cc:482] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:52.773476 14480 raft_consensus.cc:504] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:52.774149 14480 leader_election.cc:288] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:52.774622 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a875e42dcb4b6e851dd7ab8589a03b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:52.774646 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a875e42dcb4b6e851dd7ab8589a03b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:52.774849 13105 raft_consensus.cc:2454] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:52.774868 13364 raft_consensus.cc:2454] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:52.775230 13170 leader_election.cc:302] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:11:52.775489 14480 raft_consensus.cc:2781] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:52.775602 14480 raft_consensus.cc:482] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:52.775698 14480 raft_consensus.cc:3037] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:52.777807 14480 raft_consensus.cc:504] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:52.778309 14480 leader_election.cc:288] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:52.778831 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a875e42dcb4b6e851dd7ab8589a03b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:52.778894 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a875e42dcb4b6e851dd7ab8589a03b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:52.779044 13105 raft_consensus.cc:3037] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:52.779130 13364 raft_consensus.cc:3037] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:52.781080 13105 raft_consensus.cc:2454] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:52.781257 13364 raft_consensus.cc:2454] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:52.781623 13170 leader_election.cc:302] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:52.782027 14480 raft_consensus.cc:2781] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:52.782159 14480 raft_consensus.cc:686] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:11:52.782418 14480 consensus_queue.cc:237] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:52.785401 12824 catalog_manager.cc:5484] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:52.788553 14402 raft_consensus.cc:482] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:52.788700 14402 raft_consensus.cc:504] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.789341 14402 leader_election.cc:288] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:52.789816 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65842be1fe3c469dba1b57afdb5fb948" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:52.789883 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65842be1fe3c469dba1b57afdb5fb948" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:52.790055 12973 raft_consensus.cc:2454] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:52.790096 13364 raft_consensus.cc:2454] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:52.790524 13041 leader_election.cc:302] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:52.790800 14402 raft_consensus.cc:2781] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:52.790920 14402 raft_consensus.cc:482] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:52.791016 14402 raft_consensus.cc:3037] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:52.792904 14402 raft_consensus.cc:504] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.793488 14402 leader_election.cc:288] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:52.794018 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65842be1fe3c469dba1b57afdb5fb948" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:52.794232 12973 raft_consensus.cc:3037] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:52.794240 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65842be1fe3c469dba1b57afdb5fb948" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:52.794435 13364 raft_consensus.cc:3037] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:52.796528 12973 raft_consensus.cc:2454] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:52.796766 13364 raft_consensus.cc:2454] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:52.796905 13041 leader_election.cc:302] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:52.797245 14402 raft_consensus.cc:2781] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:52.797394 14402 raft_consensus.cc:686] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:52.797669 14402 consensus_queue.cc:237] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:52.800985 12824 catalog_manager.cc:5484] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
W20240501 08:11:53.160413 14509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:53.160786 14509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:11:53.165225 14462 consensus_queue.cc:1035] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:53.170506 14509 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:53.171198 14462 consensus_queue.cc:1035] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240501 08:11:53.211871 14480 consensus_queue.cc:1035] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:53.219918 14480 consensus_queue.cc:1035] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:53.220103 14402 consensus_queue.cc:1035] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:53.237214 14485 consensus_queue.cc:1035] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:53.454999 13170 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.66:35509, user_credentials={real_user=slave}} blocked reactor thread for 44385.5us
INSERT report
rows total: 2000
time total: 280.943 ms
time per row: 0.140471 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:54.586122 12831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:54.595837 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 53872527ad264dd9b4b1627ff8aa46db (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.596105 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 53872527ad264dd9b4b1627ff8aa46db (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.596567 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53872527ad264dd9b4b1627ff8aa46db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.596721 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 53872527ad264dd9b4b1627ff8aa46db (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.597151 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53872527ad264dd9b4b1627ff8aa46db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.597646 13344 tablet_service.cc:1467] Processing CreateTablet for tablet d1af72684b304cab9a7d960084ea5926 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.597908 13214 tablet_service.cc:1467] Processing CreateTablet for tablet d1af72684b304cab9a7d960084ea5926 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.598081 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1af72684b304cab9a7d960084ea5926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.598290 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1af72684b304cab9a7d960084ea5926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.600688 13213 tablet_service.cc:1467] Processing CreateTablet for tablet ec6d272070b140d9b918d182613008f4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.600728 13345 tablet_service.cc:1467] Processing CreateTablet for tablet ec6d272070b140d9b918d182613008f4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.601065 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6d272070b140d9b918d182613008f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.601070 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6d272070b140d9b918d182613008f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.602734 12953 tablet_service.cc:1467] Processing CreateTablet for tablet d1af72684b304cab9a7d960084ea5926 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.603133 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1af72684b304cab9a7d960084ea5926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.606135 13083 tablet_service.cc:1467] Processing CreateTablet for tablet ec6d272070b140d9b918d182613008f4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:54.606657 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53872527ad264dd9b4b1627ff8aa46db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.608565 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6d272070b140d9b918d182613008f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:54.622774 14580 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:54.624998 14580 tablet_bootstrap.cc:654] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.627249 14582 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:54.629700 14581 tablet_bootstrap.cc:492] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:54.630844 14582 tablet_bootstrap.cc:654] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.632203 14581 tablet_bootstrap.cc:654] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.633354 14580 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:54.633571 14580 ts_tablet_manager.cc:1367] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240501 08:11:54.634500 14580 raft_consensus.cc:348] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:54.634881 14580 raft_consensus.cc:374] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.634932 14579 tablet_bootstrap.cc:492] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:54.635094 14580 raft_consensus.cc:729] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.635692 14580 consensus_queue.cc:260] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:54.636402 14580 ts_tablet_manager.cc:1398] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:54.636787 14580 tablet_bootstrap.cc:492] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:54.636883 14582 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:54.637127 14582 ts_tablet_manager.cc:1367] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240501 08:11:54.637686 14581 tablet_bootstrap.cc:492] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:54.637895 14581 ts_tablet_manager.cc:1367] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:11:54.637993 14582 raft_consensus.cc:348] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:54.638235 14582 raft_consensus.cc:374] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.638367 14582 raft_consensus.cc:729] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.638428 14580 tablet_bootstrap.cc:654] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.638633 14582 consensus_queue.cc:260] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:54.638931 14581 raft_consensus.cc:348] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:54.639184 14581 raft_consensus.cc:374] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.639233 14582 ts_tablet_manager.cc:1398] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:54.639356 14581 raft_consensus.cc:729] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.639665 14581 consensus_queue.cc:260] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:54.639806 14582 tablet_bootstrap.cc:492] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:54.640408 14581 ts_tablet_manager.cc:1398] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:54.640764 14581 tablet_bootstrap.cc:492] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:54.641379 14582 tablet_bootstrap.cc:654] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.643950 14579 tablet_bootstrap.cc:654] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.644557 14581 tablet_bootstrap.cc:654] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.650660 14579 tablet_bootstrap.cc:492] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:54.651741 14579 ts_tablet_manager.cc:1367] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240501 08:11:54.652689 14582 tablet_bootstrap.cc:492] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:54.653035 14579 raft_consensus.cc:348] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:54.653410 14582 ts_tablet_manager.cc:1367] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.008s
I20240501 08:11:54.654273 14582 raft_consensus.cc:348] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:54.654412 14581 tablet_bootstrap.cc:492] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:54.654512 14582 raft_consensus.cc:374] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.654594 14581 ts_tablet_manager.cc:1367] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.008s
I20240501 08:11:54.654585 14582 raft_consensus.cc:729] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.654881 14582 consensus_queue.cc:260] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:54.655424 14582 ts_tablet_manager.cc:1398] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:54.655452 14581 raft_consensus.cc:348] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:54.655745 14581 raft_consensus.cc:374] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.655866 14581 raft_consensus.cc:729] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.656275 14581 consensus_queue.cc:260] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:54.657254 14581 ts_tablet_manager.cc:1398] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:54.653533 14579 raft_consensus.cc:374] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.660785 14579 raft_consensus.cc:729] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.661094 14579 consensus_queue.cc:260] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:54.661646 14579 ts_tablet_manager.cc:1398] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 08:11:54.661947 14579 tablet_bootstrap.cc:492] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:54.662846 14580 tablet_bootstrap.cc:492] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:54.663045 14580 ts_tablet_manager.cc:1367] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.013s
I20240501 08:11:54.663372 14579 tablet_bootstrap.cc:654] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.663906 14580 raft_consensus.cc:348] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:54.664218 14580 raft_consensus.cc:374] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.664775 14580 raft_consensus.cc:729] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.665095 14580 consensus_queue.cc:260] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:54.665781 14580 ts_tablet_manager.cc:1398] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:54.667047 14579 tablet_bootstrap.cc:492] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:54.667188 14579 ts_tablet_manager.cc:1367] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:54.667847 14579 raft_consensus.cc:348] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:54.668021 14579 raft_consensus.cc:374] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.668095 14579 raft_consensus.cc:729] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.668361 14579 consensus_queue.cc:260] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:54.668892 14579 ts_tablet_manager.cc:1398] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:54.669304 14579 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:54.670915 14579 tablet_bootstrap.cc:654] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:54.673018 14579 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:54.673169 14579 ts_tablet_manager.cc:1367] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:11:54.673856 14579 raft_consensus.cc:348] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:54.674022 14579 raft_consensus.cc:374] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:54.674089 14579 raft_consensus.cc:729] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:54.674242 14579 consensus_queue.cc:260] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:54.674657 14579 ts_tablet_manager.cc:1398] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:11:54.681795 14568 raft_consensus.cc:482] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:54.681955 14568 raft_consensus.cc:504] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:54.682633 14568 leader_election.cc:288] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:54.683100 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53872527ad264dd9b4b1627ff8aa46db" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:54.683131 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53872527ad264dd9b4b1627ff8aa46db" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:54.683323 13105 raft_consensus.cc:2454] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:54.683341 12973 raft_consensus.cc:2454] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:54.683764 13170 leader_election.cc:302] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:54.684036 14568 raft_consensus.cc:2781] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:54.684150 14568 raft_consensus.cc:482] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:54.684239 14568 raft_consensus.cc:3037] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.686120 14568 raft_consensus.cc:504] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:54.686714 14568 leader_election.cc:288] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:54.687147 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53872527ad264dd9b4b1627ff8aa46db" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:54.687255 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53872527ad264dd9b4b1627ff8aa46db" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:54.687355 13105 raft_consensus.cc:3037] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.687462 12973 raft_consensus.cc:3037] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.689368 13105 raft_consensus.cc:2454] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:54.689540 12973 raft_consensus.cc:2454] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:54.689759 13170 leader_election.cc:302] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:54.690088 14568 raft_consensus.cc:2781] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:54.690536 14568 raft_consensus.cc:686] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:11:54.690825 14568 consensus_queue.cc:237] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:54.693763 12832 catalog_manager.cc:5484] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:54.861456 14568 raft_consensus.cc:482] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:54.861675 14568 raft_consensus.cc:504] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:54.862402 14568 leader_election.cc:288] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:54.862967 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6d272070b140d9b918d182613008f4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:54.863067 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6d272070b140d9b918d182613008f4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:54.863231 13365 raft_consensus.cc:2454] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:54.863344 13105 raft_consensus.cc:2454] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:54.863749 13170 leader_election.cc:302] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:11:54.864091 14568 raft_consensus.cc:2781] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:54.864195 14568 raft_consensus.cc:482] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:54.864341 14568 raft_consensus.cc:3037] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.866663 14568 raft_consensus.cc:504] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:54.867332 14568 leader_election.cc:288] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:11:54.867803 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6d272070b140d9b918d182613008f4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:54.867929 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6d272070b140d9b918d182613008f4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:54.868018 13365 raft_consensus.cc:3037] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.868163 13105 raft_consensus.cc:3037] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.870005 13365 raft_consensus.cc:2454] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:54.870723 13170 leader_election.cc:302] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:11:54.870992 13105 raft_consensus.cc:2454] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:54.871029 14568 raft_consensus.cc:2781] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:54.871308 14568 raft_consensus.cc:686] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:11:54.871573 14568 consensus_queue.cc:237] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:11:54.874337 12832 catalog_manager.cc:5484] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:11:54.942761 14470 raft_consensus.cc:482] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:54.942970 14470 raft_consensus.cc:504] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:54.943658 14470 leader_election.cc:288] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:54.944098 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1af72684b304cab9a7d960084ea5926" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:54.944372 13234 raft_consensus.cc:2454] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:54.944345 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1af72684b304cab9a7d960084ea5926" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:54.944597 12973 raft_consensus.cc:2454] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:54.944999 13301 leader_election.cc:302] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:54.945319 14470 raft_consensus.cc:2781] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:54.945441 14470 raft_consensus.cc:482] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:54.945538 14470 raft_consensus.cc:3037] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.947556 14470 raft_consensus.cc:504] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:54.948091 14470 leader_election.cc:288] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:54.948591 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1af72684b304cab9a7d960084ea5926" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:54.948624 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1af72684b304cab9a7d960084ea5926" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:54.948799 13234 raft_consensus.cc:3037] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.948802 12973 raft_consensus.cc:3037] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:54.950608 12973 raft_consensus.cc:2454] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:54.950703 13234 raft_consensus.cc:2454] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:54.950984 13301 leader_election.cc:302] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:54.951325 14470 raft_consensus.cc:2781] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:54.951761 14470 raft_consensus.cc:686] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:11:54.952087 14470 consensus_queue.cc:237] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:54.954939 12832 catalog_manager.cc:5484] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:11:55.090113 14568 consensus_queue.cc:1035] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:55.099493 14568 consensus_queue.cc:1035] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:55.253315 14593 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:55.258189 14602 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:55.333307 14470 consensus_queue.cc:1035] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:55.338861 14598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:55.339316 14598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:11:55.341809 14612 consensus_queue.cc:1035] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:55.349663 14598 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
W20240501 08:11:55.595613 13171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 69029us
INSERT report
rows total: 2000
time total: 262.038 ms
time per row: 0.131019 ms
I20240501 08:11:55.767328 12831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:55.775789 13084 tablet_service.cc:1467] Processing CreateTablet for tablet b3fcc47f0a244a08b3adb00faa790da6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.776031 12954 tablet_service.cc:1467] Processing CreateTablet for tablet b3fcc47f0a244a08b3adb00faa790da6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.776270 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3fcc47f0a244a08b3adb00faa790da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.776472 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3fcc47f0a244a08b3adb00faa790da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.776666 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 76bd7c097eef44b2a35d3adc581e0cdf (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.777062 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bd7c097eef44b2a35d3adc581e0cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.781070 13344 tablet_service.cc:1467] Processing CreateTablet for tablet b3fcc47f0a244a08b3adb00faa790da6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.781493 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3fcc47f0a244a08b3adb00faa790da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.784895 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 76bd7c097eef44b2a35d3adc581e0cdf (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.785459 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bd7c097eef44b2a35d3adc581e0cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.788901 13083 tablet_service.cc:1467] Processing CreateTablet for tablet 76bd7c097eef44b2a35d3adc581e0cdf (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.789448 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76bd7c097eef44b2a35d3adc581e0cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.792429 14661 tablet_bootstrap.cc:492] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:55.794072 13085 tablet_service.cc:1467] Processing CreateTablet for tablet b178ad9ad6e54a29ad0624ef74644275 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.794605 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b178ad9ad6e54a29ad0624ef74644275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.794687 14661 tablet_bootstrap.cc:654] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.797080 13345 tablet_service.cc:1467] Processing CreateTablet for tablet b178ad9ad6e54a29ad0624ef74644275 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.797641 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b178ad9ad6e54a29ad0624ef74644275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.800335 13214 tablet_service.cc:1467] Processing CreateTablet for tablet b178ad9ad6e54a29ad0624ef74644275 (DEFAULT_TABLE table=rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:55.800737 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b178ad9ad6e54a29ad0624ef74644275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:55.810508 14661 tablet_bootstrap.cc:492] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:55.810810 14661 ts_tablet_manager.cc:1367] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.010s
I20240501 08:11:55.810920 14665 tablet_bootstrap.cc:492] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:55.811816 14661 raft_consensus.cc:348] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:55.812206 14661 raft_consensus.cc:374] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.812484 14661 raft_consensus.cc:729] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.813063 14661 consensus_queue.cc:260] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:55.813548 14662 tablet_bootstrap.cc:492] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:55.814671 14661 ts_tablet_manager.cc:1398] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:11:55.814989 14661 tablet_bootstrap.cc:492] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:55.815510 14662 tablet_bootstrap.cc:654] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.817190 14665 tablet_bootstrap.cc:654] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.817833 14661 tablet_bootstrap.cc:654] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.821509 14663 tablet_bootstrap.cc:492] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:55.821549 14662 tablet_bootstrap.cc:492] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:55.821697 14662 ts_tablet_manager.cc:1367] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240501 08:11:55.822352 14661 tablet_bootstrap.cc:492] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:55.822523 14661 ts_tablet_manager.cc:1367] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240501 08:11:55.823048 14662 raft_consensus.cc:348] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:55.823283 14663 tablet_bootstrap.cc:654] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.823302 14661 raft_consensus.cc:348] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:55.823421 14662 raft_consensus.cc:374] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.823523 14661 raft_consensus.cc:374] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.823567 14662 raft_consensus.cc:729] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.823608 14661 raft_consensus.cc:729] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.823886 14661 consensus_queue.cc:260] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:55.824442 14665 tablet_bootstrap.cc:492] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:55.824627 14661 ts_tablet_manager.cc:1398] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:55.824642 14665 ts_tablet_manager.cc:1367] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240501 08:11:55.825206 14662 consensus_queue.cc:260] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:55.825909 14663 tablet_bootstrap.cc:492] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:55.825948 14662 ts_tablet_manager.cc:1398] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 08:11:55.826088 14663 ts_tablet_manager.cc:1367] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 08:11:55.826381 14662 tablet_bootstrap.cc:492] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:55.826479 14665 raft_consensus.cc:348] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:55.826742 14665 raft_consensus.cc:374] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.826884 14665 raft_consensus.cc:729] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.826906 14663 raft_consensus.cc:348] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:55.827126 14663 raft_consensus.cc:374] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.827255 14663 raft_consensus.cc:729] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.827270 14665 consensus_queue.cc:260] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:55.827538 14663 consensus_queue.cc:260] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:55.828120 14663 ts_tablet_manager.cc:1398] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:11:55.828684 14662 tablet_bootstrap.cc:654] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.829051 14665 ts_tablet_manager.cc:1398] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:55.829396 14665 tablet_bootstrap.cc:492] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:55.829728 14663 tablet_bootstrap.cc:492] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:55.833572 14663 tablet_bootstrap.cc:654] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.833779 14665 tablet_bootstrap.cc:654] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.836160 14662 tablet_bootstrap.cc:492] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:55.836417 14662 ts_tablet_manager.cc:1367] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 08:11:55.837919 14662 raft_consensus.cc:348] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:55.838254 14662 raft_consensus.cc:374] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.838361 14662 raft_consensus.cc:729] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.838671 14662 consensus_queue.cc:260] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:55.839424 14662 ts_tablet_manager.cc:1398] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:55.839730 14662 tablet_bootstrap.cc:492] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:55.841955 14662 tablet_bootstrap.cc:654] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:55.843722 14665 tablet_bootstrap.cc:492] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:55.844014 14665 ts_tablet_manager.cc:1367] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.006s
I20240501 08:11:55.844197 14663 tablet_bootstrap.cc:492] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:55.844658 14663 ts_tablet_manager.cc:1367] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.003s
I20240501 08:11:55.845140 14665 raft_consensus.cc:348] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:55.845374 14665 raft_consensus.cc:374] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.845494 14665 raft_consensus.cc:729] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.845402 14663 raft_consensus.cc:348] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:55.845547 14662 tablet_bootstrap.cc:492] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:55.845623 14663 raft_consensus.cc:374] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.845672 14662 ts_tablet_manager.cc:1367] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20240501 08:11:55.845708 14663 raft_consensus.cc:729] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.845811 14665 consensus_queue.cc:260] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:55.845963 14663 consensus_queue.cc:260] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:55.846395 14665 ts_tablet_manager.cc:1398] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:55.846402 14662 raft_consensus.cc:348] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:55.846611 14662 raft_consensus.cc:374] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:55.846711 14662 raft_consensus.cc:729] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:55.846971 14662 consensus_queue.cc:260] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:55.847610 14662 ts_tablet_manager.cc:1398] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 08:11:55.847844 14663 ts_tablet_manager.cc:1398] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:55.892982 14460 raft_consensus.cc:482] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:55.893218 14460 raft_consensus.cc:504] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:55.893896 14460 leader_election.cc:288] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:55.894440 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bd7c097eef44b2a35d3adc581e0cdf" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:55.894482 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bd7c097eef44b2a35d3adc581e0cdf" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:55.894671 13103 raft_consensus.cc:2454] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:55.894691 13234 raft_consensus.cc:2454] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:55.895151 12910 leader_election.cc:302] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:55.895429 14460 raft_consensus.cc:2781] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:55.895555 14460 raft_consensus.cc:482] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:55.895641 14460 raft_consensus.cc:3037] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.897712 14460 raft_consensus.cc:504] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:55.898401 14460 leader_election.cc:288] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:55.898870 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bd7c097eef44b2a35d3adc581e0cdf" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:55.898932 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bd7c097eef44b2a35d3adc581e0cdf" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:55.899129 13103 raft_consensus.cc:3037] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.899195 13234 raft_consensus.cc:3037] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.901391 13103 raft_consensus.cc:2454] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:55.901552 13234 raft_consensus.cc:2454] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:55.901788 12909 leader_election.cc:302] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:55.902122 14460 raft_consensus.cc:2781] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:55.902485 14460 raft_consensus.cc:686] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:11:55.902776 14460 consensus_queue.cc:237] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:55.905527 12832 catalog_manager.cc:5484] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:11:55.916504 14460 raft_consensus.cc:482] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:55.916695 14460 raft_consensus.cc:504] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:55.917337 14460 leader_election.cc:288] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:55.917879 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3fcc47f0a244a08b3adb00faa790da6" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:55.917990 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3fcc47f0a244a08b3adb00faa790da6" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:55.918087 13103 raft_consensus.cc:2454] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:55.918215 13365 raft_consensus.cc:2454] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:55.918529 12909 leader_election.cc:302] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:55.918902 14460 raft_consensus.cc:2781] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:55.919005 14460 raft_consensus.cc:482] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:55.919098 14460 raft_consensus.cc:3037] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.921993 14460 raft_consensus.cc:504] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:55.922770 14460 leader_election.cc:288] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:55.923210 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3fcc47f0a244a08b3adb00faa790da6" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:55.923314 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3fcc47f0a244a08b3adb00faa790da6" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:55.923417 13103 raft_consensus.cc:3037] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.923528 13365 raft_consensus.cc:3037] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:55.925803 13365 raft_consensus.cc:2454] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:55.926285 12909 leader_election.cc:302] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:55.926329 13103 raft_consensus.cc:2454] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:55.926697 14460 raft_consensus.cc:2781] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:55.927179 14460 raft_consensus.cc:686] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:11:55.927453 14460 consensus_queue.cc:237] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:55.930672 12832 catalog_manager.cc:5484] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:11:56.025311 14590 raft_consensus.cc:482] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:56.025496 14590 raft_consensus.cc:504] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.026194 14590 leader_election.cc:288] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:56.026639 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b178ad9ad6e54a29ad0624ef74644275" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:56.026692 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b178ad9ad6e54a29ad0624ef74644275" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:56.026865 13365 raft_consensus.cc:2454] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:56.026942 13234 raft_consensus.cc:2454] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:56.027323 13040 leader_election.cc:302] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:56.027590 14590 raft_consensus.cc:2781] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:56.027711 14590 raft_consensus.cc:482] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:56.027824 14590 raft_consensus.cc:3037] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.029953 14590 raft_consensus.cc:504] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.030584 14590 leader_election.cc:288] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:56.031010 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b178ad9ad6e54a29ad0624ef74644275" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:56.031059 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b178ad9ad6e54a29ad0624ef74644275" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:56.031224 13365 raft_consensus.cc:3037] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.031288 13234 raft_consensus.cc:3037] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:56.033421 13365 raft_consensus.cc:2454] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:56.033881 13234 raft_consensus.cc:2454] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:56.033934 13040 leader_election.cc:302] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:56.034214 14590 raft_consensus.cc:2781] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:56.034358 14590 raft_consensus.cc:686] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:56.034611 14590 consensus_queue.cc:237] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.037612 12832 catalog_manager.cc:5484] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:56.360702 14460 consensus_queue.cc:1035] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:56.366031 14460 consensus_queue.cc:1035] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:56.458778 14680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:56.459105 14680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:56.467635 14680 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:11:56.492518 14460 consensus_queue.cc:1035] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:56.498193 14697 consensus_queue.cc:1035] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240501 08:11:56.545012 14590 consensus_queue.cc:1035] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:56.569646 14713 consensus_queue.cc:1035] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 242.376 ms
time per row: 0.121188 ms
I20240501 08:11:56.886679 12828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:56.896955 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 9348009ae1444e73b6120161309dc4a2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:56.897497 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9348009ae1444e73b6120161309dc4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.897761 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 427bc3b61a874f83bc27d9be24732e00 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:56.898197 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427bc3b61a874f83bc27d9be24732e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.901034 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 9348009ae1444e73b6120161309dc4a2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:56.901448 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9348009ae1444e73b6120161309dc4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.901382 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 427bc3b61a874f83bc27d9be24732e00 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:56.901486 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 9348009ae1444e73b6120161309dc4a2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:56.901852 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427bc3b61a874f83bc27d9be24732e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.901782 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 427bc3b61a874f83bc27d9be24732e00 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:56.902128 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427bc3b61a874f83bc27d9be24732e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.902622 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9348009ae1444e73b6120161309dc4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.905473 12952 tablet_service.cc:1467] Processing CreateTablet for tablet a412b8ae0aa648d7979fac58b50d86a6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:56.905865 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a412b8ae0aa648d7979fac58b50d86a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.917707 14751 tablet_bootstrap.cc:492] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:56.918277 13085 tablet_service.cc:1467] Processing CreateTablet for tablet a412b8ae0aa648d7979fac58b50d86a6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:56.918774 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a412b8ae0aa648d7979fac58b50d86a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.918957 13343 tablet_service.cc:1467] Processing CreateTablet for tablet a412b8ae0aa648d7979fac58b50d86a6 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:56.919385 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a412b8ae0aa648d7979fac58b50d86a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:56.921031 14751 tablet_bootstrap.cc:654] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:56.947950 14752 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:56.947945 14755 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:56.949904 14752 tablet_bootstrap.cc:654] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:56.950469 14754 tablet_bootstrap.cc:492] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:56.950585 14755 tablet_bootstrap.cc:654] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:56.952468 14754 tablet_bootstrap.cc:654] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:56.952937 14751 tablet_bootstrap.cc:492] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:56.953116 14751 ts_tablet_manager.cc:1367] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.022s
I20240501 08:11:56.953403 14755 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:56.953639 14755 ts_tablet_manager.cc:1367] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 08:11:56.953995 14751 raft_consensus.cc:348] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.954233 14751 raft_consensus.cc:374] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.954325 14751 raft_consensus.cc:729] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.954566 14751 consensus_queue.cc:260] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.955291 14751 ts_tablet_manager.cc:1398] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:11:56.956342 14751 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:56.960228 14751 tablet_bootstrap.cc:654] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:56.960860 14754 tablet_bootstrap.cc:492] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:56.961066 14754 ts_tablet_manager.cc:1367] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 08:11:56.955399 14755 raft_consensus.cc:348] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.962056 14755 raft_consensus.cc:374] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.962205 14755 raft_consensus.cc:729] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.962142 14754 raft_consensus.cc:348] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:56.962538 14754 raft_consensus.cc:374] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.962517 14755 consensus_queue.cc:260] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.962715 14754 raft_consensus.cc:729] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.963135 14754 consensus_queue.cc:260] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:56.965219 14754 ts_tablet_manager.cc:1398] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:11:56.969252 14755 ts_tablet_manager.cc:1398] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 08:11:56.974117 14755 tablet_bootstrap.cc:492] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:56.974570 14752 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:56.974745 14752 ts_tablet_manager.cc:1367] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20240501 08:11:56.975632 14752 raft_consensus.cc:348] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.975850 14752 raft_consensus.cc:374] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.975941 14752 raft_consensus.cc:729] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.976224 14752 consensus_queue.cc:260] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.976833 14752 ts_tablet_manager.cc:1398] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:56.977178 14752 tablet_bootstrap.cc:492] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:56.978902 14752 tablet_bootstrap.cc:654] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:56.980023 14755 tablet_bootstrap.cc:654] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:56.981374 14751 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:56.981613 14751 ts_tablet_manager.cc:1367] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.021s
I20240501 08:11:56.982038 14752 tablet_bootstrap.cc:492] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:56.982228 14752 ts_tablet_manager.cc:1367] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 08:11:56.982697 14751 raft_consensus.cc:348] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.983014 14751 raft_consensus.cc:374] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.983101 14751 raft_consensus.cc:729] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.983000 14752 raft_consensus.cc:348] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.983376 14752 raft_consensus.cc:374] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.983461 14752 raft_consensus.cc:729] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.983479 14751 consensus_queue.cc:260] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.983711 14752 consensus_queue.cc:260] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.984318 14751 ts_tablet_manager.cc:1398] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:11:56.984702 14751 tablet_bootstrap.cc:492] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:56.985273 14752 ts_tablet_manager.cc:1398] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:11:56.985612 14752 tablet_bootstrap.cc:492] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:56.986343 14751 tablet_bootstrap.cc:654] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:56.987478 14752 tablet_bootstrap.cc:654] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:56.988461 14751 tablet_bootstrap.cc:492] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:56.988611 14751 ts_tablet_manager.cc:1367] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 08:11:56.989651 14751 raft_consensus.cc:348] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:56.989877 14751 raft_consensus.cc:374] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.989996 14751 raft_consensus.cc:729] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.990254 14751 consensus_queue.cc:260] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:56.990950 14751 ts_tablet_manager.cc:1398] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:56.992657 14755 tablet_bootstrap.cc:492] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:56.992859 14755 ts_tablet_manager.cc:1367] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.001s
I20240501 08:11:56.992969 14752 tablet_bootstrap.cc:492] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:56.993153 14752 ts_tablet_manager.cc:1367] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:11:56.994032 14755 raft_consensus.cc:348] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.994354 14755 raft_consensus.cc:374] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.994451 14755 raft_consensus.cc:729] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.994473 14752 raft_consensus.cc:348] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:56.994768 14752 raft_consensus.cc:374] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:56.994738 14755 consensus_queue.cc:260] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:56.994891 14752 raft_consensus.cc:729] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:56.995237 14752 consensus_queue.cc:260] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:56.995766 14752 ts_tablet_manager.cc:1398] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:56.996270 14755 ts_tablet_manager.cc:1398] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:57.035218 14596 raft_consensus.cc:482] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:57.035439 14596 raft_consensus.cc:504] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:57.036103 14596 leader_election.cc:288] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:57.036764 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9348009ae1444e73b6120161309dc4a2" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:57.036789 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9348009ae1444e73b6120161309dc4a2" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:57.037020 13234 raft_consensus.cc:2454] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:57.037007 12973 raft_consensus.cc:2454] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:11:57.037526 13301 leader_election.cc:302] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:57.037936 14596 raft_consensus.cc:2781] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:57.038059 14596 raft_consensus.cc:482] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:57.038165 14596 raft_consensus.cc:3037] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.040213 14596 raft_consensus.cc:504] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:57.040798 14596 leader_election.cc:288] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:57.041280 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9348009ae1444e73b6120161309dc4a2" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:57.041379 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9348009ae1444e73b6120161309dc4a2" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:57.041457 12973 raft_consensus.cc:3037] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.041586 13234 raft_consensus.cc:3037] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.043483 12973 raft_consensus.cc:2454] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:57.043709 13234 raft_consensus.cc:2454] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:11:57.043879 13301 leader_election.cc:302] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:57.044236 14596 raft_consensus.cc:2781] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:57.044404 14596 raft_consensus.cc:686] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:11:57.044700 14596 consensus_queue.cc:237] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:57.046921 14590 raft_consensus.cc:482] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:57.047070 14590 raft_consensus.cc:504] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:57.047681 14590 leader_election.cc:288] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:57.048067 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a412b8ae0aa648d7979fac58b50d86a6" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:57.047866 12830 catalog_manager.cc:5484] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:57.048197 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a412b8ae0aa648d7979fac58b50d86a6" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:57.048255 12973 raft_consensus.cc:2454] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:57.048444 13364 raft_consensus.cc:2454] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:57.048671 13041 leader_election.cc:302] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:57.048942 14590 raft_consensus.cc:2781] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:57.049032 14590 raft_consensus.cc:482] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:57.049089 14590 raft_consensus.cc:3037] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.050920 14590 raft_consensus.cc:504] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:57.051506 14590 leader_election.cc:288] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:57.052006 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a412b8ae0aa648d7979fac58b50d86a6" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:57.051990 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a412b8ae0aa648d7979fac58b50d86a6" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:57.052207 12973 raft_consensus.cc:3037] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.052217 13364 raft_consensus.cc:3037] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.054380 12973 raft_consensus.cc:2454] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:57.054771 13041 leader_election.cc:302] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:57.054844 13364 raft_consensus.cc:2454] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:57.055037 14590 raft_consensus.cc:2781] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:57.055227 14590 raft_consensus.cc:686] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:57.055500 14590 consensus_queue.cc:237] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:57.058730 12830 catalog_manager.cc:5484] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:11:57.128161 14739 raft_consensus.cc:482] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:57.128392 14739 raft_consensus.cc:504] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:57.129052 14739 leader_election.cc:288] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:57.129549 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:57.129585 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:57.129765 13365 raft_consensus.cc:2454] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:57.129806 13234 raft_consensus.cc:2454] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:57.130200 12909 leader_election.cc:302] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:57.130472 14739 raft_consensus.cc:2781] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:57.130569 14739 raft_consensus.cc:482] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:57.130657 14739 raft_consensus.cc:3037] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.132591 14739 raft_consensus.cc:504] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:57.133214 14739 leader_election.cc:288] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:57.133678 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:57.133769 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:57.133919 13365 raft_consensus.cc:3037] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.134001 13234 raft_consensus.cc:3037] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:57.136852 13234 raft_consensus.cc:2454] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:57.136868 13365 raft_consensus.cc:2454] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:57.137282 12909 leader_election.cc:302] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:11:57.137564 14739 raft_consensus.cc:2781] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:57.137697 14739 raft_consensus.cc:686] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:11:57.137970 14739 consensus_queue.cc:237] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:57.140928 12830 catalog_manager.cc:5484] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:57.427903 14596 consensus_queue.cc:1035] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:57.434693 14716 consensus_queue.cc:1035] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:57.456454 14713 consensus_queue.cc:1035] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:57.470614 14713 consensus_queue.cc:1035] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:57.527503 14767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:57.527859 14767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:57.537307 14767 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 08:11:57.637473 14738 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:57.644533 14739 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 253.334 ms
time per row: 0.126667 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:11:58.920053 12832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:11:58.929050 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 73fafaa4edb748c9ab9d36b519ff9b07 (DEFAULT_TABLE table=rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.929355 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 73fafaa4edb748c9ab9d36b519ff9b07 (DEFAULT_TABLE table=rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.929533 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73fafaa4edb748c9ab9d36b519ff9b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.929760 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73fafaa4edb748c9ab9d36b519ff9b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.931119 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 3720cbc21425459d9f0cb91cf825e521 (DEFAULT_TABLE table=rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.931131 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 3720cbc21425459d9f0cb91cf825e521 (DEFAULT_TABLE table=rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.931514 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3720cbc21425459d9f0cb91cf825e521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.931520 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3720cbc21425459d9f0cb91cf825e521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.932754 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 4a39a9ed18924e7b88d4ae5a516105a8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.933131 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a39a9ed18924e7b88d4ae5a516105a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.936671 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 4a39a9ed18924e7b88d4ae5a516105a8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.937036 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a39a9ed18924e7b88d4ae5a516105a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.937727 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 4a39a9ed18924e7b88d4ae5a516105a8 (DEFAULT_TABLE table=rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.937774 13083 tablet_service.cc:1467] Processing CreateTablet for tablet 3720cbc21425459d9f0cb91cf825e521 (DEFAULT_TABLE table=rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.938130 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a39a9ed18924e7b88d4ae5a516105a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.938858 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3720cbc21425459d9f0cb91cf825e521. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.940280 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 73fafaa4edb748c9ab9d36b519ff9b07 (DEFAULT_TABLE table=rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:11:58.941087 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73fafaa4edb748c9ab9d36b519ff9b07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:11:58.950699 14838 tablet_bootstrap.cc:492] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:58.953088 14838 tablet_bootstrap.cc:654] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.962080 14841 tablet_bootstrap.cc:492] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:11:58.965025 14841 tablet_bootstrap.cc:654] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.967142 14837 tablet_bootstrap.cc:492] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:58.969998 14837 tablet_bootstrap.cc:654] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.972261 14842 tablet_bootstrap.cc:492] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:58.977455 14842 tablet_bootstrap.cc:654] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.977910 14837 tablet_bootstrap.cc:492] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:58.978152 14837 ts_tablet_manager.cc:1367] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240501 08:11:58.978294 14838 tablet_bootstrap.cc:492] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:58.978472 14841 tablet_bootstrap.cc:492] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:11:58.978623 14841 ts_tablet_manager.cc:1367] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20240501 08:11:58.978732 14838 ts_tablet_manager.cc:1367] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.019s
I20240501 08:11:58.979059 14837 raft_consensus.cc:348] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:58.979389 14837 raft_consensus.cc:374] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.979369 14841 raft_consensus.cc:348] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:58.979532 14837 raft_consensus.cc:729] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.979612 14841 raft_consensus.cc:374] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.979697 14841 raft_consensus.cc:729] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.979914 14841 consensus_queue.cc:260] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:58.979781 14837 consensus_queue.cc:260] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:58.980492 14841 ts_tablet_manager.cc:1398] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:11:58.980892 14837 ts_tablet_manager.cc:1398] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:58.981310 14837 tablet_bootstrap.cc:492] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:58.983513 14838 raft_consensus.cc:348] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:58.983734 14837 tablet_bootstrap.cc:654] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.983798 14838 raft_consensus.cc:374] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.983948 14838 raft_consensus.cc:729] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.984323 14838 consensus_queue.cc:260] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:58.985158 14838 ts_tablet_manager.cc:1398] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:11:58.986704 14842 tablet_bootstrap.cc:492] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:58.986865 14842 ts_tablet_manager.cc:1367] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20240501 08:11:58.985483 14838 tablet_bootstrap.cc:492] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:58.987717 14842 raft_consensus.cc:348] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:58.987898 14837 tablet_bootstrap.cc:492] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:58.987933 14842 raft_consensus.cc:374] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.988041 14842 raft_consensus.cc:729] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.988150 14837 ts_tablet_manager.cc:1367] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240501 08:11:58.988283 14842 consensus_queue.cc:260] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:58.988979 14842 ts_tablet_manager.cc:1398] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:11:58.988979 14837 raft_consensus.cc:348] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:58.989176 14837 raft_consensus.cc:374] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:58.989264 14842 tablet_bootstrap.cc:492] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:11:58.989260 14837 raft_consensus.cc:729] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:58.989563 14837 consensus_queue.cc:260] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:58.989876 14838 tablet_bootstrap.cc:654] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.990223 14837 ts_tablet_manager.cc:1398] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:11:58.990489 14837 tablet_bootstrap.cc:492] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:11:58.990868 14842 tablet_bootstrap.cc:654] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:58.992261 14837 tablet_bootstrap.cc:654] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.001269 14837 tablet_bootstrap.cc:492] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:11:59.001570 14837 ts_tablet_manager.cc:1367] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 08:11:59.002679 14837 raft_consensus.cc:348] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:59.003115 14837 raft_consensus.cc:374] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.003301 14837 raft_consensus.cc:729] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.003702 14837 consensus_queue.cc:260] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:59.004736 14837 ts_tablet_manager.cc:1398] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:59.006368 14842 tablet_bootstrap.cc:492] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:11:59.006539 14842 ts_tablet_manager.cc:1367] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20240501 08:11:59.006640 14838 tablet_bootstrap.cc:492] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:59.006853 14838 ts_tablet_manager.cc:1367] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.007s
I20240501 08:11:59.008725 14838 raft_consensus.cc:348] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:59.008925 14838 raft_consensus.cc:374] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.009006 14838 raft_consensus.cc:729] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.008925 14842 raft_consensus.cc:348] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:59.009176 14842 raft_consensus.cc:374] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.009248 14842 raft_consensus.cc:729] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.009279 14838 consensus_queue.cc:260] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:59.009488 14842 consensus_queue.cc:260] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:59.010036 14842 ts_tablet_manager.cc:1398] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:11:59.009932 14838 ts_tablet_manager.cc:1398] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:11:59.010792 14838 tablet_bootstrap.cc:492] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:11:59.012738 14838 tablet_bootstrap.cc:654] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:11:59.015322 14838 tablet_bootstrap.cc:492] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:11:59.015488 14838 ts_tablet_manager.cc:1367] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:11:59.016332 14838 raft_consensus.cc:348] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:59.016507 14838 raft_consensus.cc:374] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:11:59.016597 14838 raft_consensus.cc:729] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:11:59.016806 14838 consensus_queue.cc:260] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:59.017308 14838 ts_tablet_manager.cc:1398] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:11:59.069563 14827 raft_consensus.cc:482] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.069746 14827 raft_consensus.cc:504] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:59.070453 14827 leader_election.cc:288] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:59.070942 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3720cbc21425459d9f0cb91cf825e521" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:59.070952 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3720cbc21425459d9f0cb91cf825e521" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:59.071149 13105 raft_consensus.cc:2454] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:59.071161 13234 raft_consensus.cc:2454] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:11:59.071601 12909 leader_election.cc:302] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:59.071888 14827 raft_consensus.cc:2781] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.072016 14827 raft_consensus.cc:482] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.072113 14827 raft_consensus.cc:3037] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.074503 14827 raft_consensus.cc:504] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:59.075083 14827 leader_election.cc:288] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:11:59.075489 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3720cbc21425459d9f0cb91cf825e521" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:59.075709 13105 raft_consensus.cc:3037] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.075663 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3720cbc21425459d9f0cb91cf825e521" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:59.075881 13234 raft_consensus.cc:3037] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.078614 13105 raft_consensus.cc:2454] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:59.078658 13234 raft_consensus.cc:2454] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:11:59.079051 12909 leader_election.cc:302] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:59.079334 14827 raft_consensus.cc:2781] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.079473 14827 raft_consensus.cc:686] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:11:59.079766 14827 consensus_queue.cc:237] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:11:59.082842 12832 catalog_manager.cc:5484] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:11:59.104369 14590 raft_consensus.cc:482] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.104573 14590 raft_consensus.cc:504] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:59.105155 14590 leader_election.cc:288] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:59.105590 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73fafaa4edb748c9ab9d36b519ff9b07" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:11:59.105755 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73fafaa4edb748c9ab9d36b519ff9b07" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:11:59.105825 13234 raft_consensus.cc:2454] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:59.105970 12973 raft_consensus.cc:2454] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:11:59.106246 13041 leader_election.cc:302] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:59.106565 14590 raft_consensus.cc:2781] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.106660 14590 raft_consensus.cc:482] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.106775 14590 raft_consensus.cc:3037] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.108726 14590 raft_consensus.cc:504] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:59.109282 14590 leader_election.cc:288] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:11:59.109756 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73fafaa4edb748c9ab9d36b519ff9b07" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:11:59.109773 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73fafaa4edb748c9ab9d36b519ff9b07" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:11:59.109999 12973 raft_consensus.cc:3037] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.110002 13234 raft_consensus.cc:3037] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.112917 12973 raft_consensus.cc:2454] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:59.112903 13234 raft_consensus.cc:2454] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:11:59.113343 13041 leader_election.cc:302] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:59.113639 14590 raft_consensus.cc:2781] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.113803 14590 raft_consensus.cc:686] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:11:59.114066 14590 consensus_queue.cc:237] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:11:59.116878 12832 catalog_manager.cc:5484] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:11:59.217587 14797 raft_consensus.cc:482] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:11:59.217765 14797 raft_consensus.cc:504] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:59.218585 14797 leader_election.cc:288] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:59.219103 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a39a9ed18924e7b88d4ae5a516105a8" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:11:59.219223 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a39a9ed18924e7b88d4ae5a516105a8" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:11:59.219395 13105 raft_consensus.cc:2454] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:59.219432 13365 raft_consensus.cc:2454] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:11:59.219856 13170 leader_election.cc:302] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:11:59.220211 14797 raft_consensus.cc:2781] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:11:59.220402 14797 raft_consensus.cc:482] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:11:59.220501 14797 raft_consensus.cc:3037] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.222728 14797 raft_consensus.cc:504] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:59.223243 14797 leader_election.cc:288] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:11:59.223647 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a39a9ed18924e7b88d4ae5a516105a8" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:11:59.223698 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a39a9ed18924e7b88d4ae5a516105a8" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:11:59.223876 13105 raft_consensus.cc:3037] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.223870 13365 raft_consensus.cc:3037] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:11:59.225693 13365 raft_consensus.cc:2454] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:59.225732 13105 raft_consensus.cc:2454] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:11:59.226047 13170 leader_election.cc:302] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:11:59.226315 14797 raft_consensus.cc:2781] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:11:59.226461 14797 raft_consensus.cc:686] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:11:59.226703 14797 consensus_queue.cc:237] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:11:59.229724 12832 catalog_manager.cc:5484] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:11:59.467855 14738 consensus_queue.cc:1035] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:59.475441 14827 consensus_queue.cc:1035] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:59.526232 14590 consensus_queue.cc:1035] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:11:59.531656 14831 consensus_queue.cc:1035] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:59.626035 14855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:11:59.626415 14855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:11:59.635099 14855 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 08:11:59.791612 14797 consensus_queue.cc:1035] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:11:59.855305 13170 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.66:35509, user_credentials={real_user=slave}} blocked reactor thread for 56962.9us
I20240501 08:11:59.874657 13365 raft_consensus.cc:1261] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:11:59.880425 14797 consensus_queue.cc:1035] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 282.829 ms
time per row: 0.141415 ms
I20240501 08:11:59.974012 14909 mvcc.cc:205] Tried to move back new op lower bound from 7022801386684575744 to 7022801384354373632. Current Snapshot: MvccSnapshot[applied={T|T < 7022801386684575744}]
I20240501 08:12:00.075116 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:00.083950 12952 tablet_service.cc:1467] Processing CreateTablet for tablet a328773ef57444cb8147524107b15dba (DEFAULT_TABLE table=rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.084759 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb95e3405ae45d3bfe71fb9d552e3fe (DEFAULT_TABLE table=rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.084789 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb95e3405ae45d3bfe71fb9d552e3fe (DEFAULT_TABLE table=rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.085121 13083 tablet_service.cc:1467] Processing CreateTablet for tablet a328773ef57444cb8147524107b15dba (DEFAULT_TABLE table=rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.085234 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb95e3405ae45d3bfe71fb9d552e3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.085242 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb95e3405ae45d3bfe71fb9d552e3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.085355 13344 tablet_service.cc:1467] Processing CreateTablet for tablet a328773ef57444cb8147524107b15dba (DEFAULT_TABLE table=rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.085525 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a328773ef57444cb8147524107b15dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.085724 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a328773ef57444cb8147524107b15dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.086586 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 6fb95e3405ae45d3bfe71fb9d552e3fe (DEFAULT_TABLE table=rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.087486 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a328773ef57444cb8147524107b15dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.087771 13343 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4b83a76d7b44039ebb17245fcd8a3c (DEFAULT_TABLE table=rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.088126 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4b83a76d7b44039ebb17245fcd8a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.092459 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb95e3405ae45d3bfe71fb9d552e3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.092963 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4b83a76d7b44039ebb17245fcd8a3c (DEFAULT_TABLE table=rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.093164 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4b83a76d7b44039ebb17245fcd8a3c (DEFAULT_TABLE table=rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:00.093397 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4b83a76d7b44039ebb17245fcd8a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.093551 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4b83a76d7b44039ebb17245fcd8a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:00.108069 14922 tablet_bootstrap.cc:492] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:00.108589 14920 tablet_bootstrap.cc:492] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:00.110648 14920 tablet_bootstrap.cc:654] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.112268 14921 tablet_bootstrap.cc:492] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:00.112547 14922 tablet_bootstrap.cc:654] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.113456 14920 tablet_bootstrap.cc:492] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:00.113684 14920 ts_tablet_manager.cc:1367] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:12:00.114284 14921 tablet_bootstrap.cc:654] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.114782 14920 raft_consensus.cc:348] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.115082 14920 raft_consensus.cc:374] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.115224 14920 raft_consensus.cc:729] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.115550 14920 consensus_queue.cc:260] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.116384 14920 ts_tablet_manager.cc:1398] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:00.118189 14920 tablet_bootstrap.cc:492] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:00.119378 14925 tablet_bootstrap.cc:492] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:00.120404 14921 tablet_bootstrap.cc:492] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:00.120612 14921 ts_tablet_manager.cc:1367] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240501 08:12:00.121420 14920 tablet_bootstrap.cc:654] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.121565 14925 tablet_bootstrap.cc:654] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.121510 14921 raft_consensus.cc:348] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.121806 14921 raft_consensus.cc:374] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.121949 14921 raft_consensus.cc:729] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.122265 14921 consensus_queue.cc:260] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.122968 14921 ts_tablet_manager.cc:1398] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:00.123406 14921 tablet_bootstrap.cc:492] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:00.126130 14921 tablet_bootstrap.cc:654] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.127749 14922 tablet_bootstrap.cc:492] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:00.128057 14922 ts_tablet_manager.cc:1367] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.006s
I20240501 08:12:00.129356 14922 raft_consensus.cc:348] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:00.129748 14922 raft_consensus.cc:374] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.130025 14922 raft_consensus.cc:729] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.130466 14922 consensus_queue.cc:260] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:00.131996 14922 ts_tablet_manager.cc:1398] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:00.132340 14922 tablet_bootstrap.cc:492] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:00.134001 14922 tablet_bootstrap.cc:654] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.134374 14925 tablet_bootstrap.cc:492] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:00.134541 14925 ts_tablet_manager.cc:1367] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20240501 08:12:00.135524 14925 raft_consensus.cc:348] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.135793 14925 raft_consensus.cc:374] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.135910 14925 raft_consensus.cc:729] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.136184 14925 consensus_queue.cc:260] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.136917 14925 ts_tablet_manager.cc:1398] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:00.137297 14925 tablet_bootstrap.cc:492] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:00.140631 14925 tablet_bootstrap.cc:654] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.140709 14922 tablet_bootstrap.cc:492] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:00.140913 14922 ts_tablet_manager.cc:1367] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240501 08:12:00.141819 14921 tablet_bootstrap.cc:492] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:00.142258 14921 ts_tablet_manager.cc:1367] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.001s
I20240501 08:12:00.142150 14922 raft_consensus.cc:348] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.142498 14922 raft_consensus.cc:374] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.142624 14922 raft_consensus.cc:729] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.142987 14922 consensus_queue.cc:260] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.143184 14921 raft_consensus.cc:348] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:00.143406 14921 raft_consensus.cc:374] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.143483 14921 raft_consensus.cc:729] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.143734 14921 consensus_queue.cc:260] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:00.144408 14921 ts_tablet_manager.cc:1398] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:00.145192 14922 ts_tablet_manager.cc:1398] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:00.146018 14925 tablet_bootstrap.cc:492] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:00.146237 14920 tablet_bootstrap.cc:492] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:00.146283 14925 ts_tablet_manager.cc:1367] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240501 08:12:00.146440 14920 ts_tablet_manager.cc:1367] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.005s
I20240501 08:12:00.147076 14925 raft_consensus.cc:348] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.147289 14925 raft_consensus.cc:374] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.147230 14920 raft_consensus.cc:348] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.147362 14925 raft_consensus.cc:729] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.147469 14920 raft_consensus.cc:374] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.147575 14920 raft_consensus.cc:729] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.147599 14925 consensus_queue.cc:260] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.147986 14920 consensus_queue.cc:260] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.148573 14920 ts_tablet_manager.cc:1398] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 08:12:00.148627 14925 ts_tablet_manager.cc:1398] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:00.148905 14920 tablet_bootstrap.cc:492] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:00.150592 14920 tablet_bootstrap.cc:654] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:00.154672 14920 tablet_bootstrap.cc:492] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:00.154829 14920 ts_tablet_manager.cc:1367] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 08:12:00.155462 14920 raft_consensus.cc:348] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:00.155630 14920 raft_consensus.cc:374] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:00.155727 14920 raft_consensus.cc:729] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:00.155978 14920 consensus_queue.cc:260] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:00.156602 14920 ts_tablet_manager.cc:1398] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:00.198935 14932 raft_consensus.cc:482] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:00.199149 14932 raft_consensus.cc:504] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.200117 14932 leader_election.cc:288] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:00.200763 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a328773ef57444cb8147524107b15dba" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:00.200742 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a328773ef57444cb8147524107b15dba" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:00.201010 13105 raft_consensus.cc:2454] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:00.201020 12973 raft_consensus.cc:2454] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:00.201511 13300 leader_election.cc:302] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:00.201874 14932 raft_consensus.cc:2781] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:00.202441 14932 raft_consensus.cc:482] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:00.202809 14932 raft_consensus.cc:3037] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.205359 14932 raft_consensus.cc:504] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.206020 14932 leader_election.cc:288] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:00.206388 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a328773ef57444cb8147524107b15dba" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:00.206622 12974 raft_consensus.cc:3037] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.206637 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a328773ef57444cb8147524107b15dba" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:00.206851 13105 raft_consensus.cc:3037] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.208694 12974 raft_consensus.cc:2454] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:00.208835 13105 raft_consensus.cc:2454] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:00.209091 13301 leader_election.cc:302] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:00.209551 14932 raft_consensus.cc:2781] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:00.210048 14932 raft_consensus.cc:686] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:00.210374 14932 consensus_queue.cc:237] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.213768 12833 catalog_manager.cc:5484] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:00.248579 14827 raft_consensus.cc:482] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:00.248771 14827 raft_consensus.cc:504] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:00.249475 14827 leader_election.cc:288] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:00.249827 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4b83a76d7b44039ebb17245fcd8a3c" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:00.250022 13363 raft_consensus.cc:2454] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:00.249948 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4b83a76d7b44039ebb17245fcd8a3c" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:00.250192 13235 raft_consensus.cc:2454] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:00.250444 12909 leader_election.cc:302] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:00.250703 14827 raft_consensus.cc:2781] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:00.250808 14827 raft_consensus.cc:482] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:00.250895 14827 raft_consensus.cc:3037] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.253257 14827 raft_consensus.cc:504] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:00.253796 14827 leader_election.cc:288] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:00.254257 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4b83a76d7b44039ebb17245fcd8a3c" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:00.254439 13363 raft_consensus.cc:3037] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.254443 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4b83a76d7b44039ebb17245fcd8a3c" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:00.254639 13235 raft_consensus.cc:3037] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.256516 13235 raft_consensus.cc:2454] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:00.256657 13363 raft_consensus.cc:2454] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:00.256954 12910 leader_election.cc:302] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:00.257310 14827 raft_consensus.cc:2781] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:00.257467 14827 raft_consensus.cc:686] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:00.257745 14827 consensus_queue.cc:237] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:00.261016 12833 catalog_manager.cc:5484] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:00.460121 14905 raft_consensus.cc:482] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:00.460376 14905 raft_consensus.cc:504] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.461077 14905 leader_election.cc:288] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:00.461575 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb95e3405ae45d3bfe71fb9d552e3fe" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:00.461673 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb95e3405ae45d3bfe71fb9d552e3fe" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:00.461850 13363 raft_consensus.cc:2454] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:00.461920 13105 raft_consensus.cc:2454] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:00.462298 13170 leader_election.cc:302] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:00.462565 14905 raft_consensus.cc:2781] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:00.462688 14905 raft_consensus.cc:482] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:00.462782 14905 raft_consensus.cc:3037] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.464709 14905 raft_consensus.cc:504] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.465291 14905 leader_election.cc:288] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:00.465804 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb95e3405ae45d3bfe71fb9d552e3fe" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:00.465855 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fb95e3405ae45d3bfe71fb9d552e3fe" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:00.466059 13363 raft_consensus.cc:3037] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.466069 13105 raft_consensus.cc:3037] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:00.467939 13363 raft_consensus.cc:2454] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:00.468465 13170 leader_election.cc:302] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:00.468730 14905 raft_consensus.cc:2781] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:00.468794 13105 raft_consensus.cc:2454] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:00.469256 14905 raft_consensus.cc:686] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:00.469605 14905 consensus_queue.cc:237] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:00.473587 12833 catalog_manager.cc:5484] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:00.653235 14738 consensus_queue.cc:1035] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.656400 14932 consensus_queue.cc:1035] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.659668 14827 consensus_queue.cc:1035] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.665897 14832 consensus_queue.cc:1035] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.862779 14905 consensus_queue.cc:1035] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:00.874801 14937 consensus_queue.cc:1035] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:00.947221 14939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:00.947549 14939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:00.955471 14939 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
INSERT report
rows total: 2000
time total: 283.049 ms
time per row: 0.141524 ms
I20240501 08:12:01.399454 12832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:01.409623 13083 tablet_service.cc:1467] Processing CreateTablet for tablet eaf75b8d04534d898e233761a7f6f299 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.410117 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaf75b8d04534d898e233761a7f6f299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.410141 12952 tablet_service.cc:1467] Processing CreateTablet for tablet eaf75b8d04534d898e233761a7f6f299 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.410624 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaf75b8d04534d898e233761a7f6f299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.410678 13345 tablet_service.cc:1467] Processing CreateTablet for tablet eaf75b8d04534d898e233761a7f6f299 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.411141 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaf75b8d04534d898e233761a7f6f299. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.412199 13085 tablet_service.cc:1467] Processing CreateTablet for tablet ba940b0d8b064284bfc71952801d17c3 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.412604 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba940b0d8b064284bfc71952801d17c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.417551 13214 tablet_service.cc:1467] Processing CreateTablet for tablet ba940b0d8b064284bfc71952801d17c3 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.417976 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba940b0d8b064284bfc71952801d17c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.419641 13213 tablet_service.cc:1467] Processing CreateTablet for tablet ba6320d4451f44fb85f02e2c55393ef9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.420076 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6320d4451f44fb85f02e2c55393ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.421346 12954 tablet_service.cc:1467] Processing CreateTablet for tablet ba940b0d8b064284bfc71952801d17c3 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.421905 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba940b0d8b064284bfc71952801d17c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.424784 13343 tablet_service.cc:1467] Processing CreateTablet for tablet ba6320d4451f44fb85f02e2c55393ef9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.425345 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6320d4451f44fb85f02e2c55393ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.425665 15009 tablet_bootstrap.cc:492] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:01.427423 15009 tablet_bootstrap.cc:654] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.427819 15011 tablet_bootstrap.cc:492] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:01.430246 15011 tablet_bootstrap.cc:654] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.431859 13083 tablet_service.cc:1467] Processing CreateTablet for tablet ba6320d4451f44fb85f02e2c55393ef9 (DEFAULT_TABLE table=rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:01.432319 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6320d4451f44fb85f02e2c55393ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:01.446403 15010 tablet_bootstrap.cc:492] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:01.448038 15013 tablet_bootstrap.cc:492] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:01.448674 15010 tablet_bootstrap.cc:654] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.449921 15011 tablet_bootstrap.cc:492] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:01.450233 15011 ts_tablet_manager.cc:1367] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.006s
I20240501 08:12:01.451433 15011 raft_consensus.cc:348] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.451903 15011 raft_consensus.cc:374] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.452070 15011 raft_consensus.cc:729] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.452531 15011 consensus_queue.cc:260] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.453938 15011 ts_tablet_manager.cc:1398] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:01.454546 15011 tablet_bootstrap.cc:492] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:01.457533 15013 tablet_bootstrap.cc:654] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.462814 14905 raft_consensus.cc:482] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:01.463038 14905 raft_consensus.cc:504] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.464198 14905 leader_election.cc:288] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:01.465461 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba940b0d8b064284bfc71952801d17c3" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:01.465590 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba940b0d8b064284bfc71952801d17c3" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:01.465871 15011 tablet_bootstrap.cc:654] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
W20240501 08:12:01.466413 13171 leader_election.cc:341] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:12:01.466698 13170 leader_election.cc:341] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:01.466833 13170 leader_election.cc:302] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b; no voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187
I20240501 08:12:01.467043 15009 tablet_bootstrap.cc:492] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:01.467191 15009 ts_tablet_manager.cc:1367] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.004s
I20240501 08:12:01.467455 14905 raft_consensus.cc:2726] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:01.468431 15009 raft_consensus.cc:348] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.468838 15009 raft_consensus.cc:374] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.469014 15009 raft_consensus.cc:729] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.469401 15009 consensus_queue.cc:260] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.470163 15009 ts_tablet_manager.cc:1398] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:01.471134 15009 tablet_bootstrap.cc:492] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:01.471626 15010 tablet_bootstrap.cc:492] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:01.471674 15011 tablet_bootstrap.cc:492] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:01.471812 15010 ts_tablet_manager.cc:1367] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.005s
I20240501 08:12:01.471830 15011 ts_tablet_manager.cc:1367] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.008s
I20240501 08:12:01.472900 15010 raft_consensus.cc:348] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.472970 15011 raft_consensus.cc:348] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:01.473135 15010 raft_consensus.cc:374] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.473194 15011 raft_consensus.cc:374] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.473242 15010 raft_consensus.cc:729] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.473273 15011 raft_consensus.cc:729] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.473541 15010 consensus_queue.cc:260] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.473529 15011 consensus_queue.cc:260] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:01.474233 15011 ts_tablet_manager.cc:1398] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:01.474733 15009 tablet_bootstrap.cc:654] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.475041 15010 ts_tablet_manager.cc:1398] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:01.475389 15010 tablet_bootstrap.cc:492] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:01.477254 15010 tablet_bootstrap.cc:654] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.483278 15013 tablet_bootstrap.cc:492] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:01.483516 15013 ts_tablet_manager.cc:1367] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.005s
I20240501 08:12:01.484388 15013 raft_consensus.cc:348] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.484686 15013 raft_consensus.cc:374] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.484814 15013 raft_consensus.cc:729] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.485170 15013 consensus_queue.cc:260] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.486022 15013 ts_tablet_manager.cc:1398] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:01.486249 15010 tablet_bootstrap.cc:492] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:01.486340 15013 tablet_bootstrap.cc:492] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:01.486430 15010 ts_tablet_manager.cc:1367] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 08:12:01.487416 15010 raft_consensus.cc:348] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.487654 15010 raft_consensus.cc:374] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.487730 15010 raft_consensus.cc:729] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.487977 15010 consensus_queue.cc:260] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.489183 15013 tablet_bootstrap.cc:654] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.489797 15010 ts_tablet_manager.cc:1398] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:01.493495 15009 tablet_bootstrap.cc:492] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:01.495028 15009 ts_tablet_manager.cc:1367] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240501 08:12:01.496131 15009 raft_consensus.cc:348] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.496496 15009 raft_consensus.cc:374] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.496577 15009 raft_consensus.cc:729] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.496845 15009 consensus_queue.cc:260] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.497438 15009 ts_tablet_manager.cc:1398] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:01.497429 14827 raft_consensus.cc:482] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:01.497570 14827 raft_consensus.cc:504] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.497881 15009 tablet_bootstrap.cc:492] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:01.498287 14827 leader_election.cc:288] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:01.498808 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaf75b8d04534d898e233761a7f6f299" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:01.499014 13364 raft_consensus.cc:2454] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:01.498988 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaf75b8d04534d898e233761a7f6f299" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:01.499230 13103 raft_consensus.cc:2454] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:01.499487 12909 leader_election.cc:302] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:01.499956 15013 tablet_bootstrap.cc:492] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:01.499998 14827 raft_consensus.cc:2781] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:01.500108 14827 raft_consensus.cc:482] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:01.500227 14827 raft_consensus.cc:3037] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.503190 14827 raft_consensus.cc:504] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.503922 14827 leader_election.cc:288] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:01.504391 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaf75b8d04534d898e233761a7f6f299" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:01.504457 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaf75b8d04534d898e233761a7f6f299" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:01.504577 13103 raft_consensus.cc:3037] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.504712 13364 raft_consensus.cc:3037] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.500159 15013 ts_tablet_manager.cc:1367] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20240501 08:12:01.506249 15013 raft_consensus.cc:348] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:01.506594 15013 raft_consensus.cc:374] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.506726 15013 raft_consensus.cc:729] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.507033 15013 consensus_queue.cc:260] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:01.507390 13103 raft_consensus.cc:2454] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:01.507963 15013 ts_tablet_manager.cc:1398] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:01.507880 12909 leader_election.cc:302] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:01.508634 15000 raft_consensus.cc:2781] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:01.508816 15000 raft_consensus.cc:686] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:01.508818 13364 raft_consensus.cc:2454] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:01.509183 15000 consensus_queue.cc:237] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.510969 15009 tablet_bootstrap.cc:654] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:01.513195 15009 tablet_bootstrap.cc:492] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:01.513342 15009 ts_tablet_manager.cc:1367] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240501 08:12:01.513168 12832 catalog_manager.cc:5484] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:01.514081 15009 raft_consensus.cc:348] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:01.514248 15009 raft_consensus.cc:374] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:01.514338 15009 raft_consensus.cc:729] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:01.514549 15009 consensus_queue.cc:260] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:01.514963 15009 ts_tablet_manager.cc:1398] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:01.627746 14827 raft_consensus.cc:482] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:01.627966 14827 raft_consensus.cc:504] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.628782 14827 leader_election.cc:288] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:01.629088 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba940b0d8b064284bfc71952801d17c3" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:01.629405 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba940b0d8b064284bfc71952801d17c3" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:01.629567 13103 raft_consensus.cc:2454] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:01.629660 13235 raft_consensus.cc:2454] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:01.629976 12909 leader_election.cc:302] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:01.630337 14827 raft_consensus.cc:2781] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:01.630462 14827 raft_consensus.cc:482] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:01.630550 14827 raft_consensus.cc:3037] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.632506 14827 raft_consensus.cc:504] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.633018 14827 leader_election.cc:288] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:01.633530 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba940b0d8b064284bfc71952801d17c3" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:01.633580 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba940b0d8b064284bfc71952801d17c3" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:01.633810 13235 raft_consensus.cc:3037] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.633809 13103 raft_consensus.cc:3037] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.636034 13103 raft_consensus.cc:2454] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:01.636054 13235 raft_consensus.cc:2454] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:01.636449 12909 leader_election.cc:302] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:01.636772 14827 raft_consensus.cc:2781] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:01.636902 14827 raft_consensus.cc:686] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:01.637207 14827 consensus_queue.cc:237] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:01.640269 12827 catalog_manager.cc:5484] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:01.650928 14905 raft_consensus.cc:482] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:01.651132 14905 raft_consensus.cc:504] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:01.651813 14905 leader_election.cc:288] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:01.652381 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6320d4451f44fb85f02e2c55393ef9" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:01.652432 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6320d4451f44fb85f02e2c55393ef9" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:01.652629 13103 raft_consensus.cc:2454] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:01.652649 13364 raft_consensus.cc:2454] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:01.653031 13170 leader_election.cc:302] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:01.653275 14905 raft_consensus.cc:2781] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:01.653412 14905 raft_consensus.cc:482] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:01.653506 14905 raft_consensus.cc:3037] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.655503 14905 raft_consensus.cc:504] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:01.656019 14905 leader_election.cc:288] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:01.656488 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6320d4451f44fb85f02e2c55393ef9" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:01.656646 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6320d4451f44fb85f02e2c55393ef9" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:01.656742 13364 raft_consensus.cc:3037] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.656854 13103 raft_consensus.cc:3037] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:01.659626 13364 raft_consensus.cc:2454] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:01.659639 13103 raft_consensus.cc:2454] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:01.660123 13170 leader_election.cc:302] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:01.660456 14905 raft_consensus.cc:2781] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:01.660606 14905 raft_consensus.cc:686] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:01.660895 14905 consensus_queue.cc:237] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:01.664454 12827 catalog_manager.cc:5484] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:01.946692 14827 consensus_queue.cc:1035] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:01.954469 14827 consensus_queue.cc:1035] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:02.033996 15000 consensus_queue.cc:1035] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:02.039095 15000 consensus_queue.cc:1035] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:02.063881 15025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:02.064226 15025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:12:02.070320 14937 consensus_queue.cc:1035] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:02.072779 15025 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:02.074867 14905 consensus_queue.cc:1035] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_12'
INSERT report
rows total: 2000
time total: 228.589 ms
time per row: 0.114295 ms
I20240501 08:12:02.492214 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:02.500823 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5240d5db124f409b28fbde8cf68f8a (DEFAULT_TABLE table=rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:02.501353 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5240d5db124f409b28fbde8cf68f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:02.501598 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5240d5db124f409b28fbde8cf68f8a (DEFAULT_TABLE table=rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:02.502005 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5240d5db124f409b28fbde8cf68f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:02.502437 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5240d5db124f409b28fbde8cf68f8a (DEFAULT_TABLE table=rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:02.502842 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5240d5db124f409b28fbde8cf68f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:02.504699 12954 tablet_service.cc:1467] Processing CreateTablet for tablet a8156a46917c40e2b836a603ff70717a (DEFAULT_TABLE table=rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:02.504992 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8156a46917c40e2b836a603ff70717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:02.508265 15090 tablet_bootstrap.cc:492] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:02.509580 13085 tablet_service.cc:1467] Processing CreateTablet for tablet db4717fa6c4e45d18702741c141f6d12 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:02.510040 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4717fa6c4e45d18702741c141f6d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:02.510140 15090 tablet_bootstrap.cc:654] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:02.512828 13343 tablet_service.cc:1467] Processing CreateTablet for tablet db4717fa6c4e45d18702741c141f6d12 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:02.513324 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4717fa6c4e45d18702741c141f6d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:02.517494 13214 tablet_service.cc:1467] Processing CreateTablet for tablet a8156a46917c40e2b836a603ff70717a (DEFAULT_TABLE table=rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:02.517921 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8156a46917c40e2b836a603ff70717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:02.519264 15093 tablet_bootstrap.cc:492] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:02.520973 15090 tablet_bootstrap.cc:492] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:02.521111 15090 ts_tablet_manager.cc:1367] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
I20240501 08:12:02.521587 15093 tablet_bootstrap.cc:654] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:02.521502 12953 tablet_service.cc:1467] Processing CreateTablet for tablet db4717fa6c4e45d18702741c141f6d12 (DEFAULT_TABLE table=rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:02.521942 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4717fa6c4e45d18702741c141f6d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:02.521972 15090 raft_consensus.cc:348] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.522229 15090 raft_consensus.cc:374] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:02.522320 15090 raft_consensus.cc:729] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:02.522529 15090 consensus_queue.cc:260] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.523085 15091 tablet_bootstrap.cc:492] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:02.524555 15090 ts_tablet_manager.cc:1398] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:02.524909 15091 tablet_bootstrap.cc:654] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:02.533247 15091 tablet_bootstrap.cc:492] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:02.533573 15091 ts_tablet_manager.cc:1367] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 08:12:02.534605 15091 raft_consensus.cc:348] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.534993 15091 raft_consensus.cc:374] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:02.535140 15091 raft_consensus.cc:729] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:02.535248 15090 tablet_bootstrap.cc:492] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:02.535492 15091 consensus_queue.cc:260] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.535755 13345 tablet_service.cc:1467] Processing CreateTablet for tablet a8156a46917c40e2b836a603ff70717a (DEFAULT_TABLE table=rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:02.536160 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8156a46917c40e2b836a603ff70717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:02.536367 15091 ts_tablet_manager.cc:1398] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:02.536877 15090 tablet_bootstrap.cc:654] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:02.537905 15091 tablet_bootstrap.cc:492] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:02.539598 15091 tablet_bootstrap.cc:654] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:02.542887 15096 tablet_bootstrap.cc:492] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:02.546557 15093 tablet_bootstrap.cc:492] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:02.546895 15093 ts_tablet_manager.cc:1367] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.010s
I20240501 08:12:02.547037 15090 tablet_bootstrap.cc:492] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:02.547212 15090 ts_tablet_manager.cc:1367] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20240501 08:12:02.547678 15091 tablet_bootstrap.cc:492] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:02.547926 15091 ts_tablet_manager.cc:1367] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20240501 08:12:02.547984 15090 raft_consensus.cc:348] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.548233 15090 raft_consensus.cc:374] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:02.548336 15090 raft_consensus.cc:729] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:02.548102 15093 raft_consensus.cc:348] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:02.548614 15093 raft_consensus.cc:374] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:02.548573 15090 consensus_queue.cc:260] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.548823 15093 raft_consensus.cc:729] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:02.549120 15090 ts_tablet_manager.cc:1398] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:02.549149 15091 raft_consensus.cc:348] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:02.549437 15090 tablet_bootstrap.cc:492] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:02.549455 15091 raft_consensus.cc:374] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:02.549576 15091 raft_consensus.cc:729] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:02.549971 15091 consensus_queue.cc:260] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:02.549155 15093 consensus_queue.cc:260] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:02.551054 15090 tablet_bootstrap.cc:654] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:02.551811 15091 ts_tablet_manager.cc:1398] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:02.553236 15090 tablet_bootstrap.cc:492] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:02.553376 15090 ts_tablet_manager.cc:1367] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 08:12:02.553905 15093 ts_tablet_manager.cc:1398] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:02.554194 15093 tablet_bootstrap.cc:492] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:02.554078 15090 raft_consensus.cc:348] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:02.554277 15090 raft_consensus.cc:374] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:02.554387 15090 raft_consensus.cc:729] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:02.554620 15090 consensus_queue.cc:260] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:02.555696 15096 tablet_bootstrap.cc:654] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:02.556924 15093 tablet_bootstrap.cc:654] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:02.558068 15090 ts_tablet_manager.cc:1398] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240501 08:12:02.560806 15096 tablet_bootstrap.cc:492] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:02.561245 15096 ts_tablet_manager.cc:1367] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 08:12:02.562145 15096 raft_consensus.cc:348] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.562448 15096 raft_consensus.cc:374] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:02.562592 15096 raft_consensus.cc:729] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:02.562995 15096 consensus_queue.cc:260] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.563742 15096 ts_tablet_manager.cc:1398] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:02.565156 15093 tablet_bootstrap.cc:492] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:02.565318 15093 ts_tablet_manager.cc:1367] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240501 08:12:02.566098 15093 raft_consensus.cc:348] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.566329 15093 raft_consensus.cc:374] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:02.566401 15093 raft_consensus.cc:729] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:02.566637 15093 consensus_queue.cc:260] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.567190 15093 ts_tablet_manager.cc:1398] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:02.567489 15093 tablet_bootstrap.cc:492] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:02.568977 15093 tablet_bootstrap.cc:654] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:02.570729 15093 tablet_bootstrap.cc:492] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:02.570852 15093 ts_tablet_manager.cc:1367] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:02.571465 15093 raft_consensus.cc:348] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.571647 15093 raft_consensus.cc:374] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:02.571749 15093 raft_consensus.cc:729] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:02.571971 15093 consensus_queue.cc:260] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.572463 15093 ts_tablet_manager.cc:1398] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:02.654680 14831 raft_consensus.cc:482] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:02.654879 14831 raft_consensus.cc:504] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.656064 14831 leader_election.cc:288] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:02.656610 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4717fa6c4e45d18702741c141f6d12" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:02.656601 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4717fa6c4e45d18702741c141f6d12" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:02.656847 13364 raft_consensus.cc:2454] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:02.656867 12973 raft_consensus.cc:2454] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:02.657323 13041 leader_election.cc:302] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:02.657581 14831 raft_consensus.cc:2781] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:02.657686 14831 raft_consensus.cc:482] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:02.657775 14831 raft_consensus.cc:3037] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.661427 14831 raft_consensus.cc:504] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.662024 14831 leader_election.cc:288] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:02.662425 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4717fa6c4e45d18702741c141f6d12" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:02.662438 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4717fa6c4e45d18702741c141f6d12" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:02.662611 13364 raft_consensus.cc:3037] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.662642 12973 raft_consensus.cc:3037] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.664546 13364 raft_consensus.cc:2454] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:02.664630 12973 raft_consensus.cc:2454] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:02.665035 13041 leader_election.cc:302] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:02.665340 14831 raft_consensus.cc:2781] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:02.665508 14831 raft_consensus.cc:686] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:02.665742 14831 consensus_queue.cc:237] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.669327 12827 catalog_manager.cc:5484] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:02.707433 15100 raft_consensus.cc:482] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:02.707615 15100 raft_consensus.cc:504] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.708348 15100 leader_election.cc:288] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:02.708981 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5240d5db124f409b28fbde8cf68f8a" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:02.708961 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5240d5db124f409b28fbde8cf68f8a" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:02.709203 12973 raft_consensus.cc:2454] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:02.709204 13235 raft_consensus.cc:2454] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:02.709676 13301 leader_election.cc:302] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:02.710044 15100 raft_consensus.cc:2781] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:02.710163 15100 raft_consensus.cc:482] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:02.710250 15100 raft_consensus.cc:3037] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.713117 15100 raft_consensus.cc:504] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.713798 15100 leader_election.cc:288] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:02.714444 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5240d5db124f409b28fbde8cf68f8a" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:02.714732 12973 raft_consensus.cc:3037] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.714715 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5240d5db124f409b28fbde8cf68f8a" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:02.714936 13235 raft_consensus.cc:3037] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.717466 12973 raft_consensus.cc:2454] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:02.717990 13235 raft_consensus.cc:2454] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:02.718389 13301 leader_election.cc:302] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:02.718653 15100 raft_consensus.cc:2781] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:02.718798 15100 raft_consensus.cc:686] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:02.719154 15100 consensus_queue.cc:237] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:02.722640 12827 catalog_manager.cc:5484] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:02.753242 14827 raft_consensus.cc:482] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:02.753481 14827 raft_consensus.cc:504] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:02.754072 14827 leader_election.cc:288] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:02.754701 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8156a46917c40e2b836a603ff70717a" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:02.754701 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8156a46917c40e2b836a603ff70717a" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:02.754964 13364 raft_consensus.cc:2454] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:02.754954 13235 raft_consensus.cc:2454] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:02.755414 12910 leader_election.cc:302] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:02.755717 14827 raft_consensus.cc:2781] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:02.755800 14827 raft_consensus.cc:482] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:02.755919 14827 raft_consensus.cc:3037] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.757879 14827 raft_consensus.cc:504] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:02.758453 14827 leader_election.cc:288] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:02.758898 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8156a46917c40e2b836a603ff70717a" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:02.758958 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8156a46917c40e2b836a603ff70717a" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:02.759104 13364 raft_consensus.cc:3037] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.759177 13235 raft_consensus.cc:3037] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:02.761066 13364 raft_consensus.cc:2454] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:02.761515 13235 raft_consensus.cc:2454] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:02.761530 12909 leader_election.cc:302] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:02.761909 15032 raft_consensus.cc:2781] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:02.762048 15032 raft_consensus.cc:686] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:02.762318 15032 consensus_queue.cc:237] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:02.765841 12827 catalog_manager.cc:5484] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:03.113973 15100 consensus_queue.cc:1035] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:03.120625 14832 consensus_queue.cc:1035] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:03.139827 14831 consensus_queue.cc:1035] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:03.157997 14949 consensus_queue.cc:1035] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:03.191226 14827 consensus_queue.cc:1035] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:03.200735 14827 consensus_queue.cc:1035] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:03.216261 15107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:03.216724 15107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:03.225139 15107 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 246.296 ms
time per row: 0.123148 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:12:04.597518 12827 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:04.607528 13083 tablet_service.cc:1467] Processing CreateTablet for tablet ec027f1fcf0b4a5a87e56221f3e1e5cd (DEFAULT_TABLE table=rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.607795 13345 tablet_service.cc:1467] Processing CreateTablet for tablet ec027f1fcf0b4a5a87e56221f3e1e5cd (DEFAULT_TABLE table=rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.608089 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec027f1fcf0b4a5a87e56221f3e1e5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.608047 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 9e39ce53fb40456285a767ac836fae9d (DEFAULT_TABLE table=rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.608026 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 172dada906f54b6c842d2dd74c32203b (DEFAULT_TABLE table=rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.608203 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec027f1fcf0b4a5a87e56221f3e1e5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.608465 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e39ce53fb40456285a767ac836fae9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.609325 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 172dada906f54b6c842d2dd74c32203b (DEFAULT_TABLE table=rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.609752 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172dada906f54b6c842d2dd74c32203b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.611027 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 9e39ce53fb40456285a767ac836fae9d (DEFAULT_TABLE table=rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.611424 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e39ce53fb40456285a767ac836fae9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.612753 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 172dada906f54b6c842d2dd74c32203b (DEFAULT_TABLE table=rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.613217 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172dada906f54b6c842d2dd74c32203b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.616690 13214 tablet_service.cc:1467] Processing CreateTablet for tablet ec027f1fcf0b4a5a87e56221f3e1e5cd (DEFAULT_TABLE table=rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.617076 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec027f1fcf0b4a5a87e56221f3e1e5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.618350 15182 tablet_bootstrap.cc:492] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:04.620600 15182 tablet_bootstrap.cc:654] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.622840 15183 tablet_bootstrap.cc:492] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:04.624886 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 172dada906f54b6c842d2dd74c32203b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.625442 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 9e39ce53fb40456285a767ac836fae9d (DEFAULT_TABLE table=rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:04.625825 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e39ce53fb40456285a767ac836fae9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:04.627038 15183 tablet_bootstrap.cc:654] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.642758 15185 tablet_bootstrap.cc:492] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:04.643460 15187 tablet_bootstrap.cc:492] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:04.645148 15185 tablet_bootstrap.cc:654] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.645152 15187 tablet_bootstrap.cc:654] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.647243 15182 tablet_bootstrap.cc:492] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:04.647404 15182 ts_tablet_manager.cc:1367] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.005s
I20240501 08:12:04.647493 15187 tablet_bootstrap.cc:492] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:04.647657 15187 ts_tablet_manager.cc:1367] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:04.648312 15182 raft_consensus.cc:348] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:04.648416 15187 raft_consensus.cc:348] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:04.648607 15187 raft_consensus.cc:374] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.648649 15182 raft_consensus.cc:374] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.648754 15182 raft_consensus.cc:729] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.648746 15187 raft_consensus.cc:729] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.648972 15187 consensus_queue.cc:260] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:04.649000 15182 consensus_queue.cc:260] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:04.649371 15185 tablet_bootstrap.cc:492] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:04.649524 15185 ts_tablet_manager.cc:1367] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 08:12:04.649544 15187 ts_tablet_manager.cc:1398] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:04.650260 15183 tablet_bootstrap.cc:492] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:04.650419 15183 ts_tablet_manager.cc:1367] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.005s
I20240501 08:12:04.650323 15185 raft_consensus.cc:348] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:04.650519 15185 raft_consensus.cc:374] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.650612 15185 raft_consensus.cc:729] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.651011 15182 ts_tablet_manager.cc:1398] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:04.651268 15182 tablet_bootstrap.cc:492] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:04.651211 15185 consensus_queue.cc:260] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:04.651865 15185 ts_tablet_manager.cc:1398] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:04.652171 15185 tablet_bootstrap.cc:492] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:04.653674 15183 raft_consensus.cc:348] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:04.653913 15183 raft_consensus.cc:374] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.654026 15183 raft_consensus.cc:729] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.654137 15185 tablet_bootstrap.cc:654] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.654276 15183 consensus_queue.cc:260] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:04.657474 15183 ts_tablet_manager.cc:1398] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240501 08:12:04.658902 15187 tablet_bootstrap.cc:492] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:04.660987 15187 tablet_bootstrap.cc:654] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.661659 15182 tablet_bootstrap.cc:654] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.663728 15185 tablet_bootstrap.cc:492] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:04.663924 15185 ts_tablet_manager.cc:1367] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20240501 08:12:04.664741 15185 raft_consensus.cc:348] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:04.664997 15185 raft_consensus.cc:374] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.665081 15185 raft_consensus.cc:729] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.665388 15185 consensus_queue.cc:260] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:04.666039 15185 ts_tablet_manager.cc:1398] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:04.666527 15185 tablet_bootstrap.cc:492] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:04.669588 15185 tablet_bootstrap.cc:654] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.670590 15187 tablet_bootstrap.cc:492] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:04.670828 15187 ts_tablet_manager.cc:1367] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240501 08:12:04.671646 15187 raft_consensus.cc:348] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:04.671990 15187 raft_consensus.cc:374] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.672133 15187 raft_consensus.cc:729] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.672341 15182 tablet_bootstrap.cc:492] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:04.672513 15182 ts_tablet_manager.cc:1367] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.005s
I20240501 08:12:04.672503 15187 consensus_queue.cc:260] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:04.673339 15182 raft_consensus.cc:348] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:04.673573 15182 raft_consensus.cc:374] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.673661 15182 raft_consensus.cc:729] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.673877 15182 consensus_queue.cc:260] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:04.674139 15187 ts_tablet_manager.cc:1398] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:04.675513 15182 ts_tablet_manager.cc:1398] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:04.675810 15182 tablet_bootstrap.cc:492] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:04.678364 15185 tablet_bootstrap.cc:492] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:04.678467 15182 tablet_bootstrap.cc:654] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:04.678539 15185 ts_tablet_manager.cc:1367] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20240501 08:12:04.679275 15185 raft_consensus.cc:348] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:04.679484 15185 raft_consensus.cc:374] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.679574 15185 raft_consensus.cc:729] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.679775 15185 consensus_queue.cc:260] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:04.680277 15185 ts_tablet_manager.cc:1398] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:04.681134 15182 tablet_bootstrap.cc:492] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:04.681313 15182 ts_tablet_manager.cc:1367] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240501 08:12:04.682124 15182 raft_consensus.cc:348] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:04.682329 15182 raft_consensus.cc:374] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:04.682430 15182 raft_consensus.cc:729] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:04.682654 15182 consensus_queue.cc:260] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:04.683173 15182 ts_tablet_manager.cc:1398] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:04.718786 15100 raft_consensus.cc:482] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:04.718951 15100 raft_consensus.cc:504] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:04.719600 15100 leader_election.cc:288] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:04.720098 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec027f1fcf0b4a5a87e56221f3e1e5cd" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:04.720110 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec027f1fcf0b4a5a87e56221f3e1e5cd" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:04.720332 13234 raft_consensus.cc:2454] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:04.720394 13103 raft_consensus.cc:2454] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:04.720738 13301 leader_election.cc:302] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:04.720997 15100 raft_consensus.cc:2781] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:04.721099 15100 raft_consensus.cc:482] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:04.721191 15100 raft_consensus.cc:3037] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.723028 15100 raft_consensus.cc:504] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:04.723542 15100 leader_election.cc:288] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:04.723981 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec027f1fcf0b4a5a87e56221f3e1e5cd" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:04.723989 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec027f1fcf0b4a5a87e56221f3e1e5cd" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:04.724179 13234 raft_consensus.cc:3037] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.724176 13103 raft_consensus.cc:3037] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.726083 13103 raft_consensus.cc:2454] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:04.726259 13234 raft_consensus.cc:2454] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:04.726524 13300 leader_election.cc:302] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:04.726814 15100 raft_consensus.cc:2781] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:04.726954 15100 raft_consensus.cc:686] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:04.727216 15100 consensus_queue.cc:237] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:04.730283 12830 catalog_manager.cc:5484] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:04.751871 15177 raft_consensus.cc:482] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:04.752080 15177 raft_consensus.cc:504] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:04.752849 15177 leader_election.cc:288] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:04.753273 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e39ce53fb40456285a767ac836fae9d" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:04.753355 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e39ce53fb40456285a767ac836fae9d" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:04.753541 13103 raft_consensus.cc:2454] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:04.753537 13234 raft_consensus.cc:2454] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:04.753942 12909 leader_election.cc:302] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:04.754201 15177 raft_consensus.cc:2781] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:04.754310 15177 raft_consensus.cc:482] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:04.754380 15177 raft_consensus.cc:3037] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.756489 15177 raft_consensus.cc:504] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:04.756971 15177 leader_election.cc:288] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:04.757391 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e39ce53fb40456285a767ac836fae9d" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:04.757452 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e39ce53fb40456285a767ac836fae9d" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:04.757620 13103 raft_consensus.cc:3037] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.757622 13234 raft_consensus.cc:3037] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.760365 13234 raft_consensus.cc:2454] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:04.760408 13103 raft_consensus.cc:2454] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:04.760753 12909 leader_election.cc:302] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:04.761013 15177 raft_consensus.cc:2781] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:04.761158 15177 raft_consensus.cc:686] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:04.761442 15177 consensus_queue.cc:237] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:04.764570 12830 catalog_manager.cc:5484] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:04.973842 15196 raft_consensus.cc:482] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:04.974117 15196 raft_consensus.cc:504] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:04.974867 15196 leader_election.cc:288] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:04.975248 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172dada906f54b6c842d2dd74c32203b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:04.975394 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172dada906f54b6c842d2dd74c32203b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:04.975522 12974 raft_consensus.cc:2454] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:04.975648 13103 raft_consensus.cc:2454] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:04.976051 13170 leader_election.cc:302] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:04.976358 15196 raft_consensus.cc:2781] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:04.976477 15196 raft_consensus.cc:482] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:04.976576 15196 raft_consensus.cc:3037] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.978571 15196 raft_consensus.cc:504] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:04.979131 15196 leader_election.cc:288] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:04.979600 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172dada906f54b6c842d2dd74c32203b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:04.979637 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "172dada906f54b6c842d2dd74c32203b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:04.979812 12974 raft_consensus.cc:3037] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.979830 13103 raft_consensus.cc:3037] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:04.982255 13103 raft_consensus.cc:2454] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:04.982661 12974 raft_consensus.cc:2454] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:04.982662 13170 leader_election.cc:302] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:04.982987 15196 raft_consensus.cc:2781] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:04.983163 15196 raft_consensus.cc:686] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:04.983417 15196 consensus_queue.cc:237] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:04.986644 12830 catalog_manager.cc:5484] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:05.191463 15177 consensus_queue.cc:1035] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.197512 15202 consensus_queue.cc:1035] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.209290 15100 consensus_queue.cc:1035] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.225476 14832 consensus_queue.cc:1035] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:05.435263 15200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:05.435583 15200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:05.444928 15200 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240501 08:12:05.508056 15176 consensus_queue.cc:1035] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:05.516947 15196 consensus_queue.cc:1035] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 252.449 ms
time per row: 0.126224 ms
I20240501 08:12:05.858655 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:05.870662 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 13d9aa0c21e1405791e340cd79738256 (DEFAULT_TABLE table=rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.870723 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 89ec13cb2d2a418bba12b47f2ff4680b (DEFAULT_TABLE table=rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.871178 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d9aa0c21e1405791e340cd79738256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.871397 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ec13cb2d2a418bba12b47f2ff4680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.873234 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 13d9aa0c21e1405791e340cd79738256 (DEFAULT_TABLE table=rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.873394 13343 tablet_service.cc:1467] Processing CreateTablet for tablet 89ec13cb2d2a418bba12b47f2ff4680b (DEFAULT_TABLE table=rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.873668 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d9aa0c21e1405791e340cd79738256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.873751 13344 tablet_service.cc:1467] Processing CreateTablet for tablet d5179549829d451eb3a6e2083ef59b7f (DEFAULT_TABLE table=rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.874078 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5179549829d451eb3a6e2083ef59b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.877135 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 13d9aa0c21e1405791e340cd79738256 (DEFAULT_TABLE table=rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.877571 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d9aa0c21e1405791e340cd79738256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.880709 13083 tablet_service.cc:1467] Processing CreateTablet for tablet d5179549829d451eb3a6e2083ef59b7f (DEFAULT_TABLE table=rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.881083 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5179549829d451eb3a6e2083ef59b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.887974 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ec13cb2d2a418bba12b47f2ff4680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.888280 15267 tablet_bootstrap.cc:492] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:05.890846 15267 tablet_bootstrap.cc:654] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.899521 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 89ec13cb2d2a418bba12b47f2ff4680b (DEFAULT_TABLE table=rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.900067 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ec13cb2d2a418bba12b47f2ff4680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.900674 13214 tablet_service.cc:1467] Processing CreateTablet for tablet d5179549829d451eb3a6e2083ef59b7f (DEFAULT_TABLE table=rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:05.901049 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5179549829d451eb3a6e2083ef59b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:05.902437 15265 tablet_bootstrap.cc:492] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:05.902856 15267 tablet_bootstrap.cc:492] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:05.903122 15267 ts_tablet_manager.cc:1367] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20240501 08:12:05.904044 15267 raft_consensus.cc:348] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:05.904315 15265 tablet_bootstrap.cc:654] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.904352 15267 raft_consensus.cc:374] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.904492 15267 raft_consensus.cc:729] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.904846 15267 consensus_queue.cc:260] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:05.905644 15267 ts_tablet_manager.cc:1398] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:05.921402 15265 tablet_bootstrap.cc:492] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:05.921576 15265 ts_tablet_manager.cc:1367] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240501 08:12:05.922348 15265 raft_consensus.cc:348] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:05.922554 15265 raft_consensus.cc:374] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.922632 15265 raft_consensus.cc:729] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.922863 15265 consensus_queue.cc:260] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:05.923434 15265 ts_tablet_manager.cc:1398] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:05.923877 15265 tablet_bootstrap.cc:492] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:05.924762 15270 tablet_bootstrap.cc:492] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:05.925513 15265 tablet_bootstrap.cc:654] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.926872 15270 tablet_bootstrap.cc:654] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.939805 15265 tablet_bootstrap.cc:492] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:05.940013 15265 ts_tablet_manager.cc:1367] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240501 08:12:05.940645 15267 tablet_bootstrap.cc:492] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:05.940873 15265 raft_consensus.cc:348] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:05.941109 15265 raft_consensus.cc:374] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.941183 15265 raft_consensus.cc:729] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.941475 15265 consensus_queue.cc:260] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:05.942019 15265 ts_tablet_manager.cc:1398] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:05.942459 15267 tablet_bootstrap.cc:654] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.942416 15266 tablet_bootstrap.cc:492] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:05.946609 15266 tablet_bootstrap.cc:654] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.950800 15270 tablet_bootstrap.cc:492] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:05.951007 15267 tablet_bootstrap.cc:492] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:05.951243 15267 ts_tablet_manager.cc:1367] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 08:12:05.951403 15270 ts_tablet_manager.cc:1367] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.013s
I20240501 08:12:05.952149 15267 raft_consensus.cc:348] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:05.952488 15267 raft_consensus.cc:374] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.952466 15270 raft_consensus.cc:348] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:05.952628 15267 raft_consensus.cc:729] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.952780 15270 raft_consensus.cc:374] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.952991 15270 raft_consensus.cc:729] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.952972 15267 consensus_queue.cc:260] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:05.953579 15266 tablet_bootstrap.cc:492] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:05.953727 15267 ts_tablet_manager.cc:1398] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:05.953907 15266 ts_tablet_manager.cc:1367] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240501 08:12:05.954411 15270 consensus_queue.cc:260] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:05.954922 15266 raft_consensus.cc:348] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:05.955133 15266 raft_consensus.cc:374] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.955233 15266 raft_consensus.cc:729] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.955479 15266 consensus_queue.cc:260] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:05.957181 15266 ts_tablet_manager.cc:1398] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:05.957489 15266 tablet_bootstrap.cc:492] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:05.961263 15270 ts_tablet_manager.cc:1398] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 08:12:05.961616 15270 tablet_bootstrap.cc:492] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:05.963160 15266 tablet_bootstrap.cc:654] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.965452 15266 tablet_bootstrap.cc:492] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:05.965632 15266 ts_tablet_manager.cc:1367] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240501 08:12:05.966418 15266 raft_consensus.cc:348] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:05.966650 15270 tablet_bootstrap.cc:654] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.966668 15266 raft_consensus.cc:374] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.966737 15266 raft_consensus.cc:729] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.967011 15266 consensus_queue.cc:260] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:05.968525 15266 ts_tablet_manager.cc:1398] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:12:05.968806 15266 tablet_bootstrap.cc:492] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:05.970283 15270 tablet_bootstrap.cc:492] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:05.970484 15270 ts_tablet_manager.cc:1367] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240501 08:12:05.971268 15270 raft_consensus.cc:348] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:05.971436 15266 tablet_bootstrap.cc:654] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:05.971549 15270 raft_consensus.cc:374] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.972436 15270 raft_consensus.cc:729] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.972793 15270 consensus_queue.cc:260] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:05.973794 15266 tablet_bootstrap.cc:492] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:05.973954 15266 ts_tablet_manager.cc:1367] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:12:05.974689 15266 raft_consensus.cc:348] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:05.974875 15266 raft_consensus.cc:374] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:05.974982 15266 raft_consensus.cc:729] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:05.975199 15266 consensus_queue.cc:260] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:05.975672 15266 ts_tablet_manager.cc:1398] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:05.976090 15270 ts_tablet_manager.cc:1398] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:12:06.019917 14831 raft_consensus.cc:482] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.020148 14831 raft_consensus.cc:504] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:06.020920 14831 leader_election.cc:288] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:06.021481 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5179549829d451eb3a6e2083ef59b7f" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:06.021503 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5179549829d451eb3a6e2083ef59b7f" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:06.021724 13234 raft_consensus.cc:2454] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:06.021723 13363 raft_consensus.cc:2454] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:06.022151 13040 leader_election.cc:302] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:06.022586 14831 raft_consensus.cc:2781] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:06.022706 14831 raft_consensus.cc:482] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:06.022809 14831 raft_consensus.cc:3037] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.025089 14831 raft_consensus.cc:504] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:06.025744 14831 leader_election.cc:288] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:06.026122 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5179549829d451eb3a6e2083ef59b7f" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:06.026185 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5179549829d451eb3a6e2083ef59b7f" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:06.026311 13363 raft_consensus.cc:3037] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.026399 13234 raft_consensus.cc:3037] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.028558 13363 raft_consensus.cc:2454] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:06.028553 13234 raft_consensus.cc:2454] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:06.028951 13041 leader_election.cc:302] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:06.029403 14831 raft_consensus.cc:2781] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:06.029596 14831 raft_consensus.cc:686] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:06.029865 14831 consensus_queue.cc:237] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:06.041237 12829 catalog_manager.cc:5484] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:06.062239 14831 raft_consensus.cc:482] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.062410 14831 raft_consensus.cc:504] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:06.062989 14831 leader_election.cc:288] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:06.063443 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d9aa0c21e1405791e340cd79738256" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:06.063676 12974 raft_consensus.cc:2454] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:06.064101 13041 leader_election.cc:302] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:06.063757 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d9aa0c21e1405791e340cd79738256" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:06.064419 14831 raft_consensus.cc:2781] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:06.064517 14831 raft_consensus.cc:482] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:06.064606 14831 raft_consensus.cc:3037] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.064569 13363 raft_consensus.cc:2454] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:06.066527 14831 raft_consensus.cc:504] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:06.067086 14831 leader_election.cc:288] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:06.067510 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d9aa0c21e1405791e340cd79738256" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:06.067555 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13d9aa0c21e1405791e340cd79738256" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:06.067723 12974 raft_consensus.cc:3037] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.067770 13363 raft_consensus.cc:3037] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.069640 12974 raft_consensus.cc:2454] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:06.069720 13363 raft_consensus.cc:2454] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:06.070118 13040 leader_election.cc:302] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:06.070441 14831 raft_consensus.cc:2781] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:06.070569 14831 raft_consensus.cc:686] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:06.070817 14831 consensus_queue.cc:237] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:06.073951 12829 catalog_manager.cc:5484] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:06.136725 15177 raft_consensus.cc:482] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:06.136896 15177 raft_consensus.cc:504] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:06.137594 15177 leader_election.cc:288] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:06.138075 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ec13cb2d2a418bba12b47f2ff4680b" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:06.138202 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ec13cb2d2a418bba12b47f2ff4680b" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:06.138329 13363 raft_consensus.cc:2454] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:06.138435 13234 raft_consensus.cc:2454] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:06.138756 12909 leader_election.cc:302] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:06.139058 15177 raft_consensus.cc:2781] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:06.139191 15177 raft_consensus.cc:482] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:06.139276 15177 raft_consensus.cc:3037] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.141156 15177 raft_consensus.cc:504] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:06.141722 15177 leader_election.cc:288] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:06.142189 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ec13cb2d2a418bba12b47f2ff4680b" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:06.142182 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ec13cb2d2a418bba12b47f2ff4680b" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:06.142354 13363 raft_consensus.cc:3037] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.142392 13234 raft_consensus.cc:3037] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:06.144606 13234 raft_consensus.cc:2454] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:06.144634 13363 raft_consensus.cc:2454] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:06.145025 12910 leader_election.cc:302] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:06.145339 15177 raft_consensus.cc:2781] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:06.145474 15177 raft_consensus.cc:686] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:06.145730 15177 consensus_queue.cc:237] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:06.148994 12828 catalog_manager.cc:5484] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:06.471009 14831 consensus_queue.cc:1035] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.488051 15198 consensus_queue.cc:1035] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:06.537822 15281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:06.538187 15281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:12:06.544034 15198 consensus_queue.cc:1035] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:06.547752 15281 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:06.548509 15285 consensus_queue.cc:1035] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
I20240501 08:12:06.602412 15177 consensus_queue.cc:1035] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:06.607810 15260 consensus_queue.cc:1035] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:06.847147 12909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 51983.5us
INSERT report
rows total: 2000
time total: 264.488 ms
time per row: 0.132244 ms
I20240501 08:12:07.002669 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:07.018110 12952 tablet_service.cc:1467] Processing CreateTablet for tablet 47ee02938eee4f1ba72279b6e0b6774c (DEFAULT_TABLE table=rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.018628 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ee02938eee4f1ba72279b6e0b6774c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.018604 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 47ee02938eee4f1ba72279b6e0b6774c (DEFAULT_TABLE table=rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.019061 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ee02938eee4f1ba72279b6e0b6774c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.019471 13343 tablet_service.cc:1467] Processing CreateTablet for tablet dfdd065cc96c4ec3a248b0ae40a7fedd (DEFAULT_TABLE table=rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.019834 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdd065cc96c4ec3a248b0ae40a7fedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.020501 12953 tablet_service.cc:1467] Processing CreateTablet for tablet b568f8f64d1b4718b49e4fb55d8a0d26 (DEFAULT_TABLE table=rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.020731 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 47ee02938eee4f1ba72279b6e0b6774c (DEFAULT_TABLE table=rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.020864 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b568f8f64d1b4718b49e4fb55d8a0d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.021138 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ee02938eee4f1ba72279b6e0b6774c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.023416 12954 tablet_service.cc:1467] Processing CreateTablet for tablet dfdd065cc96c4ec3a248b0ae40a7fedd (DEFAULT_TABLE table=rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.023779 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdd065cc96c4ec3a248b0ae40a7fedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.024708 13083 tablet_service.cc:1467] Processing CreateTablet for tablet b568f8f64d1b4718b49e4fb55d8a0d26 (DEFAULT_TABLE table=rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.025079 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b568f8f64d1b4718b49e4fb55d8a0d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.028946 13215 tablet_service.cc:1467] Processing CreateTablet for tablet b568f8f64d1b4718b49e4fb55d8a0d26 (DEFAULT_TABLE table=rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.029515 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b568f8f64d1b4718b49e4fb55d8a0d26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.032735 13214 tablet_service.cc:1467] Processing CreateTablet for tablet dfdd065cc96c4ec3a248b0ae40a7fedd (DEFAULT_TABLE table=rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:07.033236 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfdd065cc96c4ec3a248b0ae40a7fedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:07.036967 15350 tablet_bootstrap.cc:492] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:07.040104 15350 tablet_bootstrap.cc:654] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.040792 15352 tablet_bootstrap.cc:492] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:07.041662 15351 tablet_bootstrap.cc:492] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:07.043324 15352 tablet_bootstrap.cc:654] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.043680 15351 tablet_bootstrap.cc:654] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.052912 15351 tablet_bootstrap.cc:492] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:07.053210 15351 ts_tablet_manager.cc:1367] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240501 08:12:07.054281 15351 raft_consensus.cc:348] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:07.054728 15351 raft_consensus.cc:374] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.054870 15351 raft_consensus.cc:729] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.055238 15351 consensus_queue.cc:260] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:07.056218 15351 ts_tablet_manager.cc:1398] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:07.056725 15351 tablet_bootstrap.cc:492] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:07.058730 15351 tablet_bootstrap.cc:654] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.062319 15352 tablet_bootstrap.cc:492] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:07.062557 15352 ts_tablet_manager.cc:1367] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.009s
I20240501 08:12:07.063500 15352 raft_consensus.cc:348] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:07.064237 15352 raft_consensus.cc:374] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.064471 15352 raft_consensus.cc:729] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.064908 15352 consensus_queue.cc:260] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:07.065776 15352 ts_tablet_manager.cc:1398] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:07.066782 15352 tablet_bootstrap.cc:492] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:07.068630 15352 tablet_bootstrap.cc:654] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.069361 15351 tablet_bootstrap.cc:492] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:07.069546 15351 ts_tablet_manager.cc:1367] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240501 08:12:07.070501 15351 raft_consensus.cc:348] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:07.070745 15351 raft_consensus.cc:374] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.070839 15351 raft_consensus.cc:729] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.071103 15350 tablet_bootstrap.cc:492] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:07.071121 15352 tablet_bootstrap.cc:492] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:07.071287 15352 ts_tablet_manager.cc:1367] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:12:07.071158 15351 consensus_queue.cc:260] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:07.071746 15351 ts_tablet_manager.cc:1398] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:07.071995 15352 raft_consensus.cc:348] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:07.072211 15352 raft_consensus.cc:374] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.072316 15352 raft_consensus.cc:729] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.072541 15352 consensus_queue.cc:260] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:07.073066 15352 ts_tablet_manager.cc:1398] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:07.073521 15350 ts_tablet_manager.cc:1367] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.021s
I20240501 08:12:07.073541 15351 tablet_bootstrap.cc:492] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:07.074482 15350 raft_consensus.cc:348] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:07.074699 15350 raft_consensus.cc:374] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.074817 15350 raft_consensus.cc:729] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.075103 15351 tablet_bootstrap.cc:654] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.075066 15350 consensus_queue.cc:260] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:07.075760 15350 ts_tablet_manager.cc:1398] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:07.076037 15350 tablet_bootstrap.cc:492] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:07.077999 15350 tablet_bootstrap.cc:654] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.088505 15356 tablet_bootstrap.cc:492] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:07.090648 15350 tablet_bootstrap.cc:492] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:07.090822 15350 ts_tablet_manager.cc:1367] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.009s
I20240501 08:12:07.091527 15356 tablet_bootstrap.cc:654] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.091636 15350 raft_consensus.cc:348] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:07.091900 15350 raft_consensus.cc:374] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.091991 15350 raft_consensus.cc:729] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.092265 15350 consensus_queue.cc:260] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:07.093675 15351 tablet_bootstrap.cc:492] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:07.093940 15351 ts_tablet_manager.cc:1367] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.006s
I20240501 08:12:07.093982 15350 ts_tablet_manager.cc:1398] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:07.094095 15356 tablet_bootstrap.cc:492] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:07.094228 15356 ts_tablet_manager.cc:1367] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:12:07.094759 15351 raft_consensus.cc:348] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:07.094839 15356 raft_consensus.cc:348] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:07.094985 15351 raft_consensus.cc:374] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.094988 15356 raft_consensus.cc:374] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.095082 15351 raft_consensus.cc:729] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.095084 15356 raft_consensus.cc:729] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.095331 15356 consensus_queue.cc:260] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:07.095328 15351 consensus_queue.cc:260] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:07.095911 15351 ts_tablet_manager.cc:1398] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:07.097096 15356 ts_tablet_manager.cc:1398] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:07.097436 15356 tablet_bootstrap.cc:492] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:07.099094 15356 tablet_bootstrap.cc:654] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:07.102562 15356 tablet_bootstrap.cc:492] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:07.102730 15356 ts_tablet_manager.cc:1367] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240501 08:12:07.103404 15356 raft_consensus.cc:348] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:07.103605 15356 raft_consensus.cc:374] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:07.103682 15356 raft_consensus.cc:729] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:07.103928 15356 consensus_queue.cc:260] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:07.104426 15356 ts_tablet_manager.cc:1398] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:07.136801 15344 raft_consensus.cc:482] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:07.136986 15344 raft_consensus.cc:504] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:07.137647 15344 leader_election.cc:288] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:07.138188 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdd065cc96c4ec3a248b0ae40a7fedd" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:07.138178 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdd065cc96c4ec3a248b0ae40a7fedd" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:07.138437 13365 raft_consensus.cc:2454] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:07.138442 12974 raft_consensus.cc:2454] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:07.138883 13171 leader_election.cc:302] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:07.139176 15344 raft_consensus.cc:2781] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:07.139276 15344 raft_consensus.cc:482] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:07.139369 15344 raft_consensus.cc:3037] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.141237 15344 raft_consensus.cc:504] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:07.141769 15344 leader_election.cc:288] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:07.142118 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdd065cc96c4ec3a248b0ae40a7fedd" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:07.142319 13365 raft_consensus.cc:3037] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.142275 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfdd065cc96c4ec3a248b0ae40a7fedd" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:07.142480 12974 raft_consensus.cc:3037] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.144699 13365 raft_consensus.cc:2454] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:07.144695 12974 raft_consensus.cc:2454] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:07.145135 13171 leader_election.cc:302] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:07.145419 15344 raft_consensus.cc:2781] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:07.145562 15344 raft_consensus.cc:686] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:07.145807 15344 consensus_queue.cc:237] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:07.149267 12830 catalog_manager.cc:5484] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:07.196503 15344 raft_consensus.cc:482] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:07.196727 15344 raft_consensus.cc:504] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:07.197454 15363 raft_consensus.cc:482] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:07.197628 15363 raft_consensus.cc:504] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:07.197852 15344 leader_election.cc:288] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:07.197852 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b568f8f64d1b4718b49e4fb55d8a0d26" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:07.198055 12974 raft_consensus.cc:2454] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:07.198177 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b568f8f64d1b4718b49e4fb55d8a0d26" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:07.198415 13103 raft_consensus.cc:2454] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:07.198498 13171 leader_election.cc:302] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:07.198776 15196 raft_consensus.cc:2781] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:07.198787 15363 leader_election.cc:288] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:07.198819 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ee02938eee4f1ba72279b6e0b6774c" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:07.198882 15196 raft_consensus.cc:482] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:07.198992 15196 raft_consensus.cc:3037] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.198987 12974 raft_consensus.cc:2454] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:07.199411 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ee02938eee4f1ba72279b6e0b6774c" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:07.199501 13301 leader_election.cc:302] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:07.199617 13103 raft_consensus.cc:2454] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:07.199795 15363 raft_consensus.cc:2781] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:07.199972 15363 raft_consensus.cc:482] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:07.200083 15363 raft_consensus.cc:3037] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.201292 15196 raft_consensus.cc:504] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:07.201931 15196 leader_election.cc:288] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:07.202067 15363 raft_consensus.cc:504] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:07.202505 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b568f8f64d1b4718b49e4fb55d8a0d26" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:07.202697 12974 raft_consensus.cc:3037] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.202834 15363 leader_election.cc:288] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:07.203065 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b568f8f64d1b4718b49e4fb55d8a0d26" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:07.203275 13103 raft_consensus.cc:3037] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.203236 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ee02938eee4f1ba72279b6e0b6774c" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:07.203424 13105 raft_consensus.cc:3037] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.205439 13103 raft_consensus.cc:2454] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:07.205550 12974 raft_consensus.cc:2454] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:07.205893 13170 leader_election.cc:302] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:07.206197 13105 raft_consensus.cc:2454] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:07.206362 15196 raft_consensus.cc:2781] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:07.206553 15196 raft_consensus.cc:686] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:07.206594 13300 leader_election.cc:302] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:07.206920 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ee02938eee4f1ba72279b6e0b6774c" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:07.206830 15196 consensus_queue.cc:237] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:07.207118 12973 raft_consensus.cc:3037] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:07.206910 15363 raft_consensus.cc:2781] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:07.207577 15363 raft_consensus.cc:686] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:07.207899 15363 consensus_queue.cc:237] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:07.209829 12973 raft_consensus.cc:2454] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:07.211923 12830 catalog_manager.cc:5484] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:07.213982 12828 catalog_manager.cc:5484] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:07.595893 15363 consensus_queue.cc:1035] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.603493 15363 consensus_queue.cc:1035] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:07.607460 15367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:07.607991 15367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:07.617602 15367 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:07.632618 15196 consensus_queue.cc:1035] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.642288 15344 consensus_queue.cc:1035] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_16'
I20240501 08:12:07.675151 15344 consensus_queue.cc:1035] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:07.688683 15344 consensus_queue.cc:1035] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:07.851329 13171 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 44901.2us
INSERT report
rows total: 2000
time total: 284.586 ms
time per row: 0.142293 ms
I20240501 08:12:08.060802 12826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:08.070632 13083 tablet_service.cc:1467] Processing CreateTablet for tablet faa5b3517ffc490cab68d09bbe2d5c0e (DEFAULT_TABLE table=rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.071115 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa5b3517ffc490cab68d09bbe2d5c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.071425 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 5741e5458ccd4af6a3972495ccf40b8f (DEFAULT_TABLE table=rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.071826 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5741e5458ccd4af6a3972495ccf40b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.072357 13343 tablet_service.cc:1467] Processing CreateTablet for tablet 5741e5458ccd4af6a3972495ccf40b8f (DEFAULT_TABLE table=rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.072333 13345 tablet_service.cc:1467] Processing CreateTablet for tablet faa5b3517ffc490cab68d09bbe2d5c0e (DEFAULT_TABLE table=rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.072818 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa5b3517ffc490cab68d09bbe2d5c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.073015 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 74d1a1cd0bf94d0d9a87674d41838d2d (DEFAULT_TABLE table=rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.073022 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 74d1a1cd0bf94d0d9a87674d41838d2d (DEFAULT_TABLE table=rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.073395 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d1a1cd0bf94d0d9a87674d41838d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.073433 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d1a1cd0bf94d0d9a87674d41838d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.076515 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5741e5458ccd4af6a3972495ccf40b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.076717 13214 tablet_service.cc:1467] Processing CreateTablet for tablet faa5b3517ffc490cab68d09bbe2d5c0e (DEFAULT_TABLE table=rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.077143 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa5b3517ffc490cab68d09bbe2d5c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.081341 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 5741e5458ccd4af6a3972495ccf40b8f (DEFAULT_TABLE table=rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.081781 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5741e5458ccd4af6a3972495ccf40b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.084692 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 74d1a1cd0bf94d0d9a87674d41838d2d (DEFAULT_TABLE table=rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:08.085105 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d1a1cd0bf94d0d9a87674d41838d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:08.093465 15434 tablet_bootstrap.cc:492] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:08.093680 15436 tablet_bootstrap.cc:492] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:08.095535 15436 tablet_bootstrap.cc:654] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.096181 15435 tablet_bootstrap.cc:492] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:08.097033 15434 tablet_bootstrap.cc:654] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.098779 15435 tablet_bootstrap.cc:654] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.105792 15434 tablet_bootstrap.cc:492] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:08.106127 15434 ts_tablet_manager.cc:1367] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240501 08:12:08.107445 15434 raft_consensus.cc:348] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.108079 15434 raft_consensus.cc:374] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.108270 15434 raft_consensus.cc:729] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.108734 15434 consensus_queue.cc:260] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.109431 15434 ts_tablet_manager.cc:1398] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:08.110874 15434 tablet_bootstrap.cc:492] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:08.113127 15436 tablet_bootstrap.cc:492] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:08.113374 15436 ts_tablet_manager.cc:1367] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.005s
I20240501 08:12:08.113459 15434 tablet_bootstrap.cc:654] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.114030 15433 tablet_bootstrap.cc:492] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:08.114356 15436 raft_consensus.cc:348] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.114708 15436 raft_consensus.cc:374] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.114854 15436 raft_consensus.cc:729] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.115175 15436 consensus_queue.cc:260] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.116906 15436 ts_tablet_manager.cc:1398] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:08.117218 15436 tablet_bootstrap.cc:492] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:08.117988 15433 tablet_bootstrap.cc:654] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.120389 15436 tablet_bootstrap.cc:654] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.120816 15434 tablet_bootstrap.cc:492] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:08.121045 15434 ts_tablet_manager.cc:1367] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240501 08:12:08.121822 15434 raft_consensus.cc:348] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.122103 15434 raft_consensus.cc:374] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.122220 15434 raft_consensus.cc:729] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.122501 15434 consensus_queue.cc:260] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.122887 15435 tablet_bootstrap.cc:492] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:08.123031 15434 ts_tablet_manager.cc:1398] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:08.123054 15435 ts_tablet_manager.cc:1367] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.004s
I20240501 08:12:08.123658 15435 raft_consensus.cc:348] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.123829 15435 raft_consensus.cc:374] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.123919 15435 raft_consensus.cc:729] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.124157 15435 consensus_queue.cc:260] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.124284 15433 tablet_bootstrap.cc:492] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:08.124470 15433 ts_tablet_manager.cc:1367] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240501 08:12:08.125139 15433 raft_consensus.cc:348] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.125358 15433 raft_consensus.cc:374] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.125448 15433 raft_consensus.cc:729] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.125722 15433 consensus_queue.cc:260] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.125802 15435 ts_tablet_manager.cc:1398] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:08.126452 15433 ts_tablet_manager.cc:1398] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:08.126624 15436 tablet_bootstrap.cc:492] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:08.126785 15433 tablet_bootstrap.cc:492] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:08.126860 15436 ts_tablet_manager.cc:1367] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240501 08:12:08.127743 15436 raft_consensus.cc:348] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.128094 15436 raft_consensus.cc:374] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.128218 15436 raft_consensus.cc:729] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.128381 15433 tablet_bootstrap.cc:654] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.128566 15436 consensus_queue.cc:260] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.130761 15433 tablet_bootstrap.cc:492] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:08.130981 15433 ts_tablet_manager.cc:1367] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:08.131585 15436 ts_tablet_manager.cc:1398] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:12:08.131929 15436 tablet_bootstrap.cc:492] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:08.132651 15433 raft_consensus.cc:348] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.133008 15433 raft_consensus.cc:374] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.133141 15433 raft_consensus.cc:729] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.133527 15433 consensus_queue.cc:260] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.133927 15436 tablet_bootstrap.cc:654] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.134824 15433 ts_tablet_manager.cc:1398] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:08.135152 15433 tablet_bootstrap.cc:492] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:08.138088 15433 tablet_bootstrap.cc:654] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:08.140583 15385 raft_consensus.cc:482] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:08.140765 15385 raft_consensus.cc:504] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.141539 15385 leader_election.cc:288] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:08.141749 15436 tablet_bootstrap.cc:492] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:08.141804 15433 tablet_bootstrap.cc:492] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:08.141955 15433 ts_tablet_manager.cc:1367] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 08:12:08.141973 15436 ts_tablet_manager.cc:1367] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 08:12:08.142398 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5741e5458ccd4af6a3972495ccf40b8f" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:08.142665 13105 raft_consensus.cc:2454] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:08.142645 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5741e5458ccd4af6a3972495ccf40b8f" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:08.142788 15436 raft_consensus.cc:348] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.142788 15433 raft_consensus.cc:348] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.143035 15436 raft_consensus.cc:374] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.143108 15433 raft_consensus.cc:374] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:08.143126 15436 raft_consensus.cc:729] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.143215 15433 raft_consensus.cc:729] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:08.143164 13170 leader_election.cc:302] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:08.143451 15385 raft_consensus.cc:2781] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:08.143365 15436 consensus_queue.cc:260] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.142851 13365 raft_consensus.cc:2454] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:08.143566 15385 raft_consensus.cc:482] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:08.143648 15385 raft_consensus.cc:3037] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.143426 15433 consensus_queue.cc:260] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.143936 15436 ts_tablet_manager.cc:1398] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:08.144112 15433 ts_tablet_manager.cc:1398] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:08.146386 15385 raft_consensus.cc:504] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.147385 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5741e5458ccd4af6a3972495ccf40b8f" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:08.147639 13105 raft_consensus.cc:3037] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.148555 15385 leader_election.cc:288] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:08.149819 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5741e5458ccd4af6a3972495ccf40b8f" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:08.149986 13365 raft_consensus.cc:3037] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.150019 13105 raft_consensus.cc:2454] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:08.150398 13170 leader_election.cc:302] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:08.150681 15385 raft_consensus.cc:2781] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:08.150976 15385 raft_consensus.cc:686] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:08.151276 15385 consensus_queue.cc:237] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.152772 13365 raft_consensus.cc:2454] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:08.154647 12827 catalog_manager.cc:5484] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:08.213514 15363 raft_consensus.cc:482] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:08.213714 15363 raft_consensus.cc:504] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.214655 15198 raft_consensus.cc:482] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:08.214704 15363 leader_election.cc:288] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:08.214854 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d1a1cd0bf94d0d9a87674d41838d2d" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:08.214823 15198 raft_consensus.cc:504] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.215082 12972 raft_consensus.cc:2454] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:08.215502 13301 leader_election.cc:302] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:08.215476 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d1a1cd0bf94d0d9a87674d41838d2d" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:08.215716 13105 raft_consensus.cc:2454] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:08.215574 15198 leader_election.cc:288] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:08.216012 15363 raft_consensus.cc:2781] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:08.216130 15363 raft_consensus.cc:482] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:08.216233 15363 raft_consensus.cc:3037] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.216159 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d1a1cd0bf94d0d9a87674d41838d2d" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:08.216375 12972 raft_consensus.cc:2454] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:08.216698 13041 leader_election.cc:302] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:08.216934 15198 raft_consensus.cc:2781] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:08.217029 15198 raft_consensus.cc:482] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:08.217115 15198 raft_consensus.cc:3037] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.218407 15363 raft_consensus.cc:504] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.219242 15198 raft_consensus.cc:504] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.219439 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d1a1cd0bf94d0d9a87674d41838d2d" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:08.216012 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d1a1cd0bf94d0d9a87674d41838d2d" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:08.219658 12972 raft_consensus.cc:3037] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.219830 13365 raft_consensus.cc:2379] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3fefd4dca2d41678187047f06c10187 in current term 1: Already voted for candidate 4ecb724cb4764a53988fde00407db4c0 in this term.
I20240501 08:12:08.219862 15198 leader_election.cc:288] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:08.220460 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d1a1cd0bf94d0d9a87674d41838d2d" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:08.220604 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d1a1cd0bf94d0d9a87674d41838d2d" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:08.220814 13364 raft_consensus.cc:2379] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e3fefd4dca2d41678187047f06c10187 in current term 1: Already voted for candidate 4ecb724cb4764a53988fde00407db4c0 in this term.
I20240501 08:12:08.221230 13040 leader_election.cc:302] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3fefd4dca2d41678187047f06c10187; no voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:08.221514 15198 raft_consensus.cc:2726] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:12:08.221772 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74d1a1cd0bf94d0d9a87674d41838d2d" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:08.222034 13105 raft_consensus.cc:2379] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ecb724cb4764a53988fde00407db4c0 in current term 1: Already voted for candidate e3fefd4dca2d41678187047f06c10187 in this term.
I20240501 08:12:08.222374 15363 leader_election.cc:288] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:08.222712 12972 raft_consensus.cc:2454] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:08.223057 13301 leader_election.cc:302] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters: e3fefd4dca2d41678187047f06c10187
I20240501 08:12:08.223358 15363 raft_consensus.cc:2781] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:08.223536 15363 raft_consensus.cc:686] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:08.223798 15363 consensus_queue.cc:237] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.227375 12827 catalog_manager.cc:5484] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:08.237556 15363 raft_consensus.cc:482] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:08.237713 15363 raft_consensus.cc:504] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.238297 15363 leader_election.cc:288] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:08.238822 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa5b3517ffc490cab68d09bbe2d5c0e" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:08.238842 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa5b3517ffc490cab68d09bbe2d5c0e" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:08.239039 13105 raft_consensus.cc:2454] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:08.239079 13235 raft_consensus.cc:2454] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:08.239454 13300 leader_election.cc:302] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:08.239794 15363 raft_consensus.cc:2781] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:08.239910 15363 raft_consensus.cc:482] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:08.240010 15363 raft_consensus.cc:3037] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.243008 15363 raft_consensus.cc:504] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.244364 15363 leader_election.cc:288] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:08.245092 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa5b3517ffc490cab68d09bbe2d5c0e" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:08.245169 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faa5b3517ffc490cab68d09bbe2d5c0e" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:08.245287 13105 raft_consensus.cc:3037] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.245355 13235 raft_consensus.cc:3037] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:08.247218 13105 raft_consensus.cc:2454] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:08.247243 13235 raft_consensus.cc:2454] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:08.247637 13301 leader_election.cc:302] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:08.247928 15363 raft_consensus.cc:2781] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:08.248065 15363 raft_consensus.cc:686] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:08.248376 15363 consensus_queue.cc:237] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:08.251639 12827 catalog_manager.cc:5484] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:08.547827 15417 consensus_queue.cc:1035] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.565397 15385 consensus_queue.cc:1035] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:08.632067 15449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:08.632584 15449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:12:08.633934 15363 consensus_queue.cc:1035] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.638833 15363 consensus_queue.cc:1035] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:08.642735 15449 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240501 08:12:08.710320 15363 consensus_queue.cc:1035] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:08.779449 15363 consensus_queue.cc:1035] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 418.525 ms
time per row: 0.209262 ms
I20240501 08:12:09.223348 12828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:09.232354 12953 tablet_service.cc:1467] Processing CreateTablet for tablet ad1faf7987004fe29256bdbf993ce2e5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.232757 13084 tablet_service.cc:1467] Processing CreateTablet for tablet ad1faf7987004fe29256bdbf993ce2e5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.232908 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1faf7987004fe29256bdbf993ce2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.233213 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1faf7987004fe29256bdbf993ce2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.233407 13345 tablet_service.cc:1467] Processing CreateTablet for tablet bf1baafa5548431180ca3bc8d35607ad (DEFAULT_TABLE table=rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.233825 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1baafa5548431180ca3bc8d35607ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.234084 13215 tablet_service.cc:1467] Processing CreateTablet for tablet ad1faf7987004fe29256bdbf993ce2e5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.234472 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1faf7987004fe29256bdbf993ce2e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.237063 13085 tablet_service.cc:1467] Processing CreateTablet for tablet d7cd8edcdc5c4f22b5ba33224887d027 (DEFAULT_TABLE table=rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.237478 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7cd8edcdc5c4f22b5ba33224887d027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.237481 12952 tablet_service.cc:1467] Processing CreateTablet for tablet bf1baafa5548431180ca3bc8d35607ad (DEFAULT_TABLE table=rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.237897 12952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1baafa5548431180ca3bc8d35607ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.240914 13214 tablet_service.cc:1467] Processing CreateTablet for tablet bf1baafa5548431180ca3bc8d35607ad (DEFAULT_TABLE table=rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.241382 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf1baafa5548431180ca3bc8d35607ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.242156 12954 tablet_service.cc:1467] Processing CreateTablet for tablet d7cd8edcdc5c4f22b5ba33224887d027 (DEFAULT_TABLE table=rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.242676 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7cd8edcdc5c4f22b5ba33224887d027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.248200 13213 tablet_service.cc:1467] Processing CreateTablet for tablet d7cd8edcdc5c4f22b5ba33224887d027 (DEFAULT_TABLE table=rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:09.250792 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7cd8edcdc5c4f22b5ba33224887d027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:09.272401 15516 tablet_bootstrap.cc:492] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:09.273902 15515 tablet_bootstrap.cc:492] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:09.278931 15515 tablet_bootstrap.cc:654] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.279350 15516 tablet_bootstrap.cc:654] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.285038 15513 tablet_bootstrap.cc:492] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:09.286897 15514 tablet_bootstrap.cc:492] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:09.287886 15513 tablet_bootstrap.cc:654] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.288213 15514 tablet_bootstrap.cc:654] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.290199 15514 tablet_bootstrap.cc:492] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:09.290346 15514 ts_tablet_manager.cc:1367] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:09.291107 15514 raft_consensus.cc:348] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:09.291328 15514 raft_consensus.cc:374] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.291435 15514 raft_consensus.cc:729] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.291662 15514 consensus_queue.cc:260] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:09.292330 15514 ts_tablet_manager.cc:1398] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:09.292795 15514 tablet_bootstrap.cc:492] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:09.292996 15516 tablet_bootstrap.cc:492] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:09.293156 15516 ts_tablet_manager.cc:1367] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.004s
I20240501 08:12:09.293956 15516 raft_consensus.cc:348] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:09.294163 15516 raft_consensus.cc:374] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.294258 15516 raft_consensus.cc:729] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.294415 15514 tablet_bootstrap.cc:654] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.294497 15516 consensus_queue.cc:260] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:09.295256 15513 tablet_bootstrap.cc:492] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:09.295477 15513 ts_tablet_manager.cc:1367] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 08:12:09.296310 15514 tablet_bootstrap.cc:492] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:09.296272 15513 raft_consensus.cc:348] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:09.296483 15514 ts_tablet_manager.cc:1367] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 08:12:09.296489 15513 raft_consensus.cc:374] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.296573 15513 raft_consensus.cc:729] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.297127 15516 ts_tablet_manager.cc:1398] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:09.297389 15513 consensus_queue.cc:260] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:09.298305 15514 raft_consensus.cc:348] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:09.298511 15514 raft_consensus.cc:374] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.298584 15514 raft_consensus.cc:729] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.299083 15516 tablet_bootstrap.cc:492] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:09.300807 15514 consensus_queue.cc:260] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:09.301216 15515 tablet_bootstrap.cc:492] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:09.301364 15515 ts_tablet_manager.cc:1367] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.004s
I20240501 08:12:09.301440 15514 ts_tablet_manager.cc:1398] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:12:09.301766 15514 tablet_bootstrap.cc:492] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:09.302057 15515 raft_consensus.cc:348] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:09.302258 15515 raft_consensus.cc:374] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.302326 15515 raft_consensus.cc:729] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.302532 15515 consensus_queue.cc:260] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:09.303162 15515 ts_tablet_manager.cc:1398] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:09.304661 15514 tablet_bootstrap.cc:654] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.305459 15516 tablet_bootstrap.cc:654] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.306677 15513 ts_tablet_manager.cc:1398] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 08:12:09.309296 15513 tablet_bootstrap.cc:492] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:09.311137 15513 tablet_bootstrap.cc:654] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.313282 15514 tablet_bootstrap.cc:492] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:09.313439 15514 ts_tablet_manager.cc:1367] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240501 08:12:09.313457 15516 tablet_bootstrap.cc:492] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:09.313613 15516 ts_tablet_manager.cc:1367] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240501 08:12:09.314185 15514 raft_consensus.cc:348] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:09.314285 15516 raft_consensus.cc:348] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:09.314416 15514 raft_consensus.cc:374] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.314468 15516 raft_consensus.cc:374] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.314517 15514 raft_consensus.cc:729] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.314618 15516 raft_consensus.cc:729] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.314783 15514 consensus_queue.cc:260] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:09.314858 15516 consensus_queue.cc:260] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:09.315482 15516 ts_tablet_manager.cc:1398] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:09.317956 15514 ts_tablet_manager.cc:1398] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:09.321444 15513 tablet_bootstrap.cc:492] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:09.321669 15513 ts_tablet_manager.cc:1367] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20240501 08:12:09.322391 15513 raft_consensus.cc:348] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:09.322559 15513 raft_consensus.cc:374] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.322613 15513 raft_consensus.cc:729] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.322788 15513 consensus_queue.cc:260] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:09.323237 15513 ts_tablet_manager.cc:1398] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:12:09.323576 15513 tablet_bootstrap.cc:492] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:09.325048 15513 tablet_bootstrap.cc:654] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:09.326968 15513 tablet_bootstrap.cc:492] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:09.327121 15513 ts_tablet_manager.cc:1367] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:12:09.327893 15513 raft_consensus.cc:348] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:09.328125 15513 raft_consensus.cc:374] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:09.328227 15513 raft_consensus.cc:729] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:09.328505 15513 consensus_queue.cc:260] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:09.329082 15513 ts_tablet_manager.cc:1398] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:09.338794 15417 raft_consensus.cc:482] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:09.338997 15417 raft_consensus.cc:504] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:09.339591 15417 leader_election.cc:288] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:09.340200 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1faf7987004fe29256bdbf993ce2e5" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:09.340209 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1faf7987004fe29256bdbf993ce2e5" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:09.340488 12972 raft_consensus.cc:2454] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:09.340495 13103 raft_consensus.cc:2454] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:09.340974 13171 leader_election.cc:302] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:09.341265 15417 raft_consensus.cc:2781] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:09.341358 15417 raft_consensus.cc:482] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:09.341468 15417 raft_consensus.cc:3037] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.343716 15417 raft_consensus.cc:504] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:09.344256 15417 leader_election.cc:288] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:09.344620 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1faf7987004fe29256bdbf993ce2e5" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:09.344753 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1faf7987004fe29256bdbf993ce2e5" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:09.344846 12972 raft_consensus.cc:3037] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.344972 13103 raft_consensus.cc:3037] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.347692 12972 raft_consensus.cc:2454] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:09.347703 13103 raft_consensus.cc:2454] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:09.348098 13171 leader_election.cc:302] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:09.348456 15417 raft_consensus.cc:2781] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:09.348657 15417 raft_consensus.cc:686] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:09.348912 15417 consensus_queue.cc:237] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:09.353545 12828 catalog_manager.cc:5484] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:09.407109 15417 raft_consensus.cc:482] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:09.407317 15417 raft_consensus.cc:504] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:09.408421 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1baafa5548431180ca3bc8d35607ad" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:09.408485 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1baafa5548431180ca3bc8d35607ad" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:09.408668 13365 raft_consensus.cc:2454] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:09.408728 12972 raft_consensus.cc:2454] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:09.409001 15417 leader_election.cc:288] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:09.409303 13170 leader_election.cc:302] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:09.409598 15417 raft_consensus.cc:2781] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:09.409716 15417 raft_consensus.cc:482] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:09.409797 15417 raft_consensus.cc:3037] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.412756 15417 raft_consensus.cc:504] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:09.413403 15417 leader_election.cc:288] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:09.413890 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1baafa5548431180ca3bc8d35607ad" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:09.414115 12972 raft_consensus.cc:3037] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.416479 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf1baafa5548431180ca3bc8d35607ad" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:09.416687 13365 raft_consensus.cc:3037] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.416867 12972 raft_consensus.cc:2454] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:09.417300 13171 leader_election.cc:302] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:09.417608 15417 raft_consensus.cc:2781] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:09.417770 15417 raft_consensus.cc:686] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:09.418125 15417 consensus_queue.cc:237] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:09.419322 13365 raft_consensus.cc:2454] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:09.422619 12824 catalog_manager.cc:5484] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:09.500926 15285 raft_consensus.cc:482] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:09.501132 15285 raft_consensus.cc:504] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:09.501740 15285 leader_election.cc:288] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:09.502306 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7cd8edcdc5c4f22b5ba33224887d027" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:09.502311 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7cd8edcdc5c4f22b5ba33224887d027" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:09.502573 12972 raft_consensus.cc:2454] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:09.502579 13235 raft_consensus.cc:2454] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:09.502966 13041 leader_election.cc:302] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:09.503258 15285 raft_consensus.cc:2781] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:09.503347 15285 raft_consensus.cc:482] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:09.503450 15285 raft_consensus.cc:3037] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.505800 15285 raft_consensus.cc:504] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:09.506372 15285 leader_election.cc:288] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:09.506852 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7cd8edcdc5c4f22b5ba33224887d027" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:09.506865 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7cd8edcdc5c4f22b5ba33224887d027" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:09.507068 12972 raft_consensus.cc:3037] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.507059 13235 raft_consensus.cc:3037] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:09.509649 12972 raft_consensus.cc:2454] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:09.509807 13235 raft_consensus.cc:2454] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:09.510048 13041 leader_election.cc:302] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:09.510380 15285 raft_consensus.cc:2781] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:09.510646 15285 raft_consensus.cc:686] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:09.510921 15285 consensus_queue.cc:237] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:09.514457 12824 catalog_manager.cc:5484] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:09.799955 15417 consensus_queue.cc:1035] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.805488 15385 consensus_queue.cc:1035] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.874028 15500 consensus_queue.cc:1035] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.897786 15534 consensus_queue.cc:1035] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:09.947197 15530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:09.947652 15530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:09.960197 15530 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:12:09.979825 15531 consensus_queue.cc:1035] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:09.985046 15198 consensus_queue.cc:1035] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
INSERT report
rows total: 2000
time total: 223.909 ms
time per row: 0.111954 ms
I20240501 08:12:10.343279 12824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:10.352342 13345 tablet_service.cc:1467] Processing CreateTablet for tablet d4ba66f68a134d09bc53d840ba8191d6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.352847 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ba66f68a134d09bc53d840ba8191d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.352886 13085 tablet_service.cc:1467] Processing CreateTablet for tablet d4ba66f68a134d09bc53d840ba8191d6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.353197 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 7883772f69cd4a18a2a6d568adbdeb61 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.353312 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ba66f68a134d09bc53d840ba8191d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.353610 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7883772f69cd4a18a2a6d568adbdeb61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.356690 12953 tablet_service.cc:1467] Processing CreateTablet for tablet d4ba66f68a134d09bc53d840ba8191d6 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.357141 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ba66f68a134d09bc53d840ba8191d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.358268 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 911cc47749484e88882c0efa75fc6396 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.358650 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911cc47749484e88882c0efa75fc6396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.359253 15591 tablet_bootstrap.cc:492] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:10.360939 15591 tablet_bootstrap.cc:654] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.361097 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 7883772f69cd4a18a2a6d568adbdeb61 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.361505 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7883772f69cd4a18a2a6d568adbdeb61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.363816 15593 tablet_bootstrap.cc:492] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:10.365636 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 7883772f69cd4a18a2a6d568adbdeb61 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.366044 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7883772f69cd4a18a2a6d568adbdeb61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.368610 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 911cc47749484e88882c0efa75fc6396 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.368999 15593 tablet_bootstrap.cc:654] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.369012 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911cc47749484e88882c0efa75fc6396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.370134 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 911cc47749484e88882c0efa75fc6396 (DEFAULT_TABLE table=rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:10.370666 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911cc47749484e88882c0efa75fc6396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:10.390779 15591 tablet_bootstrap.cc:492] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:10.391042 15591 ts_tablet_manager.cc:1367] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.015s
I20240501 08:12:10.392113 15591 raft_consensus.cc:348] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:10.392460 15591 raft_consensus.cc:374] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.392619 15591 raft_consensus.cc:729] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.392923 15591 consensus_queue.cc:260] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:10.393507 15595 tablet_bootstrap.cc:492] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:10.393596 15591 ts_tablet_manager.cc:1398] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:10.394980 15591 tablet_bootstrap.cc:492] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:10.396546 15591 tablet_bootstrap.cc:654] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.397902 15595 tablet_bootstrap.cc:654] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.403405 15595 tablet_bootstrap.cc:492] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:10.403569 15595 ts_tablet_manager.cc:1367] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240501 08:12:10.404232 15595 raft_consensus.cc:348] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:10.404471 15595 raft_consensus.cc:374] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.404577 15595 raft_consensus.cc:729] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.404807 15595 consensus_queue.cc:260] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:10.405400 15595 ts_tablet_manager.cc:1398] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:10.405694 15595 tablet_bootstrap.cc:492] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:10.407277 15595 tablet_bootstrap.cc:654] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.410756 15596 tablet_bootstrap.cc:492] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:10.412623 15596 tablet_bootstrap.cc:654] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.413570 15591 tablet_bootstrap.cc:492] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:10.413758 15591 ts_tablet_manager.cc:1367] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20240501 08:12:10.418833 15593 tablet_bootstrap.cc:492] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:10.419019 15593 ts_tablet_manager.cc:1367] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.055s user 0.023s sys 0.012s
I20240501 08:12:10.418848 15591 raft_consensus.cc:348] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:10.419164 15591 raft_consensus.cc:374] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.419267 15591 raft_consensus.cc:729] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.419651 15591 consensus_queue.cc:260] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:10.419818 15593 raft_consensus.cc:348] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:10.420024 15593 raft_consensus.cc:374] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.420128 15593 raft_consensus.cc:729] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.420514 15591 ts_tablet_manager.cc:1398] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240501 08:12:10.420428 15593 consensus_queue.cc:260] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:10.421022 15593 ts_tablet_manager.cc:1398] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:10.421031 15591 tablet_bootstrap.cc:492] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:10.421720 15593 tablet_bootstrap.cc:492] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:10.422632 15591 tablet_bootstrap.cc:654] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.423378 15593 tablet_bootstrap.cc:654] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.425082 15595 tablet_bootstrap.cc:492] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:10.425307 15595 ts_tablet_manager.cc:1367] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20240501 08:12:10.426131 15595 raft_consensus.cc:348] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:10.426374 15595 raft_consensus.cc:374] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.426441 15595 raft_consensus.cc:729] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.426744 15595 consensus_queue.cc:260] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:10.427351 15595 ts_tablet_manager.cc:1398] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:10.429788 15596 tablet_bootstrap.cc:492] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:10.429966 15596 ts_tablet_manager.cc:1367] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.003s
I20240501 08:12:10.430145 15593 tablet_bootstrap.cc:492] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:10.430331 15593 ts_tablet_manager.cc:1367] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240501 08:12:10.430689 15591 tablet_bootstrap.cc:492] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:10.430825 15591 ts_tablet_manager.cc:1367] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20240501 08:12:10.431190 15593 raft_consensus.cc:348] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:10.431437 15593 raft_consensus.cc:374] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.431516 15593 raft_consensus.cc:729] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.431583 15591 raft_consensus.cc:348] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:10.431784 15591 raft_consensus.cc:374] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.431872 15591 raft_consensus.cc:729] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.431774 15593 consensus_queue.cc:260] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:10.432122 15591 consensus_queue.cc:260] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:10.432438 15593 ts_tablet_manager.cc:1398] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 08:12:10.432771 15591 ts_tablet_manager.cc:1398] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:10.432967 15596 raft_consensus.cc:348] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:10.433152 15596 raft_consensus.cc:374] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.433254 15596 raft_consensus.cc:729] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.433477 15596 consensus_queue.cc:260] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:10.434103 15596 ts_tablet_manager.cc:1398] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:10.434506 15596 tablet_bootstrap.cc:492] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:10.437283 15596 tablet_bootstrap.cc:654] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:10.442960 15596 tablet_bootstrap.cc:492] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:10.443230 15596 ts_tablet_manager.cc:1367] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240501 08:12:10.444499 15596 raft_consensus.cc:348] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:10.444731 15596 raft_consensus.cc:374] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:10.444804 15596 raft_consensus.cc:729] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:10.445060 15596 consensus_queue.cc:260] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:10.445734 15596 ts_tablet_manager.cc:1398] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:10.464490 15260 raft_consensus.cc:482] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:10.464733 15260 raft_consensus.cc:504] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:10.465515 15260 leader_election.cc:288] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:10.466482 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ba66f68a134d09bc53d840ba8191d6" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:10.466699 13103 raft_consensus.cc:2454] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:10.467860 12909 leader_election.cc:302] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:10.468179 15260 raft_consensus.cc:2781] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:10.468271 15260 raft_consensus.cc:482] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:10.468394 15260 raft_consensus.cc:3037] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.469647 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ba66f68a134d09bc53d840ba8191d6" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:10.469874 13365 raft_consensus.cc:2454] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:10.471526 15260 raft_consensus.cc:504] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:10.472646 15260 leader_election.cc:288] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:10.472806 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ba66f68a134d09bc53d840ba8191d6" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:10.473014 13365 raft_consensus.cc:3037] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.475687 13365 raft_consensus.cc:2454] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:10.476169 12909 leader_election.cc:302] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:10.476467 15260 raft_consensus.cc:2781] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:10.476536 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ba66f68a134d09bc53d840ba8191d6" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:10.476763 13103 raft_consensus.cc:3037] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.476760 15260 raft_consensus.cc:686] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:10.477035 15260 consensus_queue.cc:237] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:10.479698 13103 raft_consensus.cc:2454] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:10.482343 12824 catalog_manager.cc:5484] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:10.512800 15500 raft_consensus.cc:482] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:10.512991 15500 raft_consensus.cc:504] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:10.513588 15500 leader_election.cc:288] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:10.514124 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7883772f69cd4a18a2a6d568adbdeb61" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:10.514166 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7883772f69cd4a18a2a6d568adbdeb61" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:10.514355 12973 raft_consensus.cc:2454] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:10.514439 13365 raft_consensus.cc:2454] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:10.514873 13171 leader_election.cc:302] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:10.515192 15500 raft_consensus.cc:2781] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:10.515296 15500 raft_consensus.cc:482] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:10.515380 15500 raft_consensus.cc:3037] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.517375 15500 raft_consensus.cc:504] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:10.517961 15500 leader_election.cc:288] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:10.518337 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7883772f69cd4a18a2a6d568adbdeb61" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:10.518404 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7883772f69cd4a18a2a6d568adbdeb61" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:10.518549 13365 raft_consensus.cc:3037] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.518615 12973 raft_consensus.cc:3037] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.520727 13365 raft_consensus.cc:2454] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:10.520938 12973 raft_consensus.cc:2454] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:10.521142 13170 leader_election.cc:302] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:10.521437 15500 raft_consensus.cc:2781] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:10.521649 15500 raft_consensus.cc:686] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:10.521898 15500 consensus_queue.cc:237] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:10.525300 12824 catalog_manager.cc:5484] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:10.605680 15198 raft_consensus.cc:482] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:10.605870 15198 raft_consensus.cc:504] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:10.606489 15198 leader_election.cc:288] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:10.607031 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911cc47749484e88882c0efa75fc6396" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:10.607031 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911cc47749484e88882c0efa75fc6396" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:10.607267 13365 raft_consensus.cc:2454] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:10.607267 13235 raft_consensus.cc:2454] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:10.607722 13040 leader_election.cc:302] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:10.608032 15198 raft_consensus.cc:2781] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:10.608141 15198 raft_consensus.cc:482] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:10.608238 15198 raft_consensus.cc:3037] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.610435 15198 raft_consensus.cc:504] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:10.610980 15198 leader_election.cc:288] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:10.611534 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911cc47749484e88882c0efa75fc6396" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:10.611760 13235 raft_consensus.cc:3037] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.612640 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911cc47749484e88882c0efa75fc6396" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:10.612838 13365 raft_consensus.cc:3037] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:10.614285 13235 raft_consensus.cc:2454] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:10.614635 13041 leader_election.cc:302] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:10.615021 15198 raft_consensus.cc:2781] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:10.615092 13365 raft_consensus.cc:2454] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:10.615319 15198 raft_consensus.cc:686] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:10.615698 15198 consensus_queue.cc:237] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:10.619108 12824 catalog_manager.cc:5484] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:10.938300 15385 consensus_queue.cc:1035] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:10.954821 15500 consensus_queue.cc:1035] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.002082 15260 consensus_queue.cc:1035] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.013602 15260 consensus_queue.cc:1035] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:11.019559 15607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:11.020021 15607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:11.029628 15607 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240501 08:12:11.161101 15198 consensus_queue.cc:1035] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:11.197549 15198 consensus_queue.cc:1035] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 280.215 ms
time per row: 0.140108 ms
I20240501 08:12:11.506567 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.69:42897
--local_ip_for_outbound_sockets=127.6.56.69
--tserver_master_addrs=127.6.56.126:44861
--webserver_port=34933
--webserver_interface=127.6.56.69
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:12:11.871683 15673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:11.872105 15673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:11.872247 15673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:11.884951 15673 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:12:11.885118 15673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:11.885298 15673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.69
I20240501 08:12:11.897010 15673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.69:42897
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.56.69
--webserver_port=34933
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.69
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:11.898586 15673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:11.900349 15673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:11.914206 15680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:11.915017 15679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:11.915012 15682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:11.915735 15673 server_base.cc:1034] running on GCE node
I20240501 08:12:11.916411 15673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:11.917439 15673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:11.918640 15673 hybrid_clock.cc:648] HybridClock initialized: now 1714551131918576 us; error 57 us; skew 500 ppm
I20240501 08:12:11.918987 15673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:11.927019 15673 webserver.cc:460] Webserver started at http://127.6.56.69:34933/ using document root <none> and password file <none>
I20240501 08:12:11.927815 15673 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:11.927953 15673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:11.933877 15673 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 08:12:11.936923 15688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:11.938302 15673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 08:12:11.938462 15673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "982f49fc87f149439cda0de2261b6bfd"
format_stamp: "Formatted at 2024-05-01 08:11:37 on dist-test-slave-f1nk"
I20240501 08:12:11.939260 15673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:11.961669 15673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:11.962507 15673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:11.962823 15673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:11.963774 15673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:11.965543 15673 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:11.965629 15673 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:11.965747 15673 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:11.965807 15673 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:12.000639 15673 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.69:42897
I20240501 08:12:12.000658 15800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.69:42897 every 8 connection(s)
I20240501 08:12:12.002511 15673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:12:12.012017 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 15673
I20240501 08:12:12.012878 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.70:43301
--local_ip_for_outbound_sockets=127.6.56.70
--tserver_master_addrs=127.6.56.126:44861
--webserver_port=34153
--webserver_interface=127.6.56.70
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:12:12.016638 15801 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:12:12.017232 15801 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:12.018054 15801 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:12:12.020578 12832 ts_manager.cc:194] Re-registered known tserver with Master: 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:12.021418 12832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.69:52369
W20240501 08:12:12.381467 15805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:12.381795 15805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:12.381920 15805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:12.390753 15805 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:12:12.390910 15805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:12.391067 15805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.70
I20240501 08:12:12.401423 15805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.70:43301
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.56.70
--webserver_port=34153
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.70
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:12.402812 15805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:12.404945 15805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:12.418327 15813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:12.419114 15816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:12.419622 15814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:12.420830 15805 server_base.cc:1034] running on GCE node
I20240501 08:12:12.421468 15805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:12.422717 15805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:12.423990 15805 hybrid_clock.cc:648] HybridClock initialized: now 1714551132423740 us; error 242 us; skew 500 ppm
I20240501 08:12:12.424388 15805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:12.426635 15805 webserver.cc:460] Webserver started at http://127.6.56.70:34153/ using document root <none> and password file <none>
I20240501 08:12:12.427423 15805 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:12.427552 15805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:12.434093 15805 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20240501 08:12:12.436985 15822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:12.438362 15805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:12.438511 15805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "4d516916eeac468db514eeb928f06190"
format_stamp: "Formatted at 2024-05-01 08:11:37 on dist-test-slave-f1nk"
I20240501 08:12:12.439352 15805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:12.473470 15805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:12.474555 15805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:12.474895 15805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:12.476045 15805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:12.477859 15805 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:12.477936 15805 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:12.478032 15805 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:12.478077 15805 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:12.511749 15805 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.70:43301
I20240501 08:12:12.511786 15937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.70:43301 every 8 connection(s)
I20240501 08:12:12.513566 15805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:12:12.518826 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 15805
I20240501 08:12:12.519493 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.71:36997
--local_ip_for_outbound_sockets=127.6.56.71
--tserver_master_addrs=127.6.56.126:44861
--webserver_port=33273
--webserver_interface=127.6.56.71
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:12:12.527189 15938 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:12:12.527523 15938 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:12.528262 15938 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:12:12.529784 12832 ts_manager.cc:194] Re-registered known tserver with Master: 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:12.530613 12832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.70:57049
W20240501 08:12:12.890136 15942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:12.890426 15942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:12.890554 15942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:12.898376 15942 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:12:12.898546 15942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:12.898722 15942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.71
I20240501 08:12:12.909168 15942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.71:36997
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.56.71
--webserver_port=33273
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.71
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:12.910753 15942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:12.912719 15942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:12.928375 15948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:12.929353 15950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:12.930636 15947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:12.934590 15942 server_base.cc:1034] running on GCE node
I20240501 08:12:12.935323 15942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:12.936357 15942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:12.937563 15942 hybrid_clock.cc:648] HybridClock initialized: now 1714551132937519 us; error 34 us; skew 500 ppm
I20240501 08:12:12.937891 15942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:12.940519 15942 webserver.cc:460] Webserver started at http://127.6.56.71:33273/ using document root <none> and password file <none>
I20240501 08:12:12.941347 15942 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:12.941499 15942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:12.947420 15942 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20240501 08:12:12.950346 15956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:12.951766 15942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:12.951930 15942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5dd722c770d34d6c9154a87917117161"
format_stamp: "Formatted at 2024-05-01 08:11:38 on dist-test-slave-f1nk"
I20240501 08:12:12.953049 15942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:12.972802 15942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:12.973672 15942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:12.973974 15942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:12.974944 15942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:12.976644 15942 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:12.976729 15942 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:12.976871 15942 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:12.976938 15942 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:13.016592 15942 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.71:36997
I20240501 08:12:13.016654 16069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.71:36997 every 8 connection(s)
I20240501 08:12:13.018151 15942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:12:13.024475 15801 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
I20240501 08:12:13.027082 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 15942
I20240501 08:12:13.027820 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.72:45383
--local_ip_for_outbound_sockets=127.6.56.72
--tserver_master_addrs=127.6.56.126:44861
--webserver_port=46621
--webserver_interface=127.6.56.72
--builtin_ntp_servers=127.6.56.84:40899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:12:13.033512 16070 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:12:13.033875 16070 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:13.034883 16070 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:12:13.036548 12830 ts_manager.cc:194] Re-registered known tserver with Master: 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:13.037588 12830 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.71:42421
W20240501 08:12:13.397415 16073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:13.397727 16073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:12:13.397851 16073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:13.405894 16073 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:12:13.406031 16073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:12:13.406188 16073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.72
I20240501 08:12:13.416750 16073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:40899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.72:45383
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.56.72
--webserver_port=46621
--tserver_master_addrs=127.6.56.126:44861
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.72
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:12:13.418130 16073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:12:13.420236 16073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:12:13.434178 16081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:13.434404 16080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:12:13.435314 16083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:12:13.436793 16073 server_base.cc:1034] running on GCE node
I20240501 08:12:13.437413 16073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:12:13.438388 16073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:12:13.439586 16073 hybrid_clock.cc:648] HybridClock initialized: now 1714551133439531 us; error 41 us; skew 500 ppm
I20240501 08:12:13.439919 16073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:12:13.443414 16073 webserver.cc:460] Webserver started at http://127.6.56.72:46621/ using document root <none> and password file <none>
I20240501 08:12:13.444219 16073 fs_manager.cc:359] Metadata directory not provided
I20240501 08:12:13.444389 16073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:12:13.450881 16073 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20240501 08:12:13.454492 16089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:13.456058 16073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:13.456243 16073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
format_stamp: "Formatted at 2024-05-01 08:11:38 on dist-test-slave-f1nk"
I20240501 08:12:13.457156 16073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:12:13.476226 16073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:12:13.477236 16073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:12:13.477537 16073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:12:13.478543 16073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:12:13.480211 16073 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:12:13.480311 16073 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:13.480456 16073 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:12:13.480511 16073 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:12:13.522704 16073 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.72:45383
I20240501 08:12:13.522738 16202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.72:45383 every 8 connection(s)
I20240501 08:12:13.524206 16073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:12:13.524786 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 16073
I20240501 08:12:13.533931 15938 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
I20240501 08:12:13.541054 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:13.549329 16203 heartbeater.cc:347] Connected to a master server at 127.6.56.126:44861
I20240501 08:12:13.549715 16203 heartbeater.cc:464] Registering TS with master...
I20240501 08:12:13.550762 16203 heartbeater.cc:511] Master 127.6.56.126:44861 requested a full tablet report, sending...
I20240501 08:12:13.552758 12832 ts_manager.cc:194] Re-registered known tserver with Master: 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:13.554026 12832 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.72:54659
I20240501 08:12:13.573463 13085 tablet_service.cc:1467] Processing CreateTablet for tablet b3e6757fba154ce4915f4d447970ffd4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cd07cc53a527402f9e3860fb5443bda2]), partition=
I20240501 08:12:13.573468 13345 tablet_service.cc:1467] Processing CreateTablet for tablet b3e6757fba154ce4915f4d447970ffd4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cd07cc53a527402f9e3860fb5443bda2]), partition=
I20240501 08:12:13.573873 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e6757fba154ce4915f4d447970ffd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.573884 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e6757fba154ce4915f4d447970ffd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.592051 16217 tablet_bootstrap.cc:492] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:13.592308 15736 tablet_service.cc:1467] Processing CreateTablet for tablet b3e6757fba154ce4915f4d447970ffd4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cd07cc53a527402f9e3860fb5443bda2]), partition=
I20240501 08:12:13.593578 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e6757fba154ce4915f4d447970ffd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:13.594408 16217 tablet_bootstrap.cc:654] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.609066 16218 tablet_bootstrap.cc:492] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:13.610838 16218 tablet_bootstrap.cc:654] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.621012 16217 tablet_bootstrap.cc:492] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:13.621253 16217 ts_tablet_manager.cc:1367] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.004s
I20240501 08:12:13.622089 16217 raft_consensus.cc:348] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:13.622385 16217 raft_consensus.cc:374] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.622501 16217 raft_consensus.cc:729] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.622949 16217 consensus_queue.cc:260] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:13.623756 16218 tablet_bootstrap.cc:492] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:13.624027 16218 ts_tablet_manager.cc:1367] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240501 08:12:13.624912 16218 raft_consensus.cc:348] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:13.625186 16218 raft_consensus.cc:374] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.625313 16218 raft_consensus.cc:729] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.625494 16217 ts_tablet_manager.cc:1398] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:12:13.625660 16218 consensus_queue.cc:260] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:13.626324 16218 ts_tablet_manager.cc:1398] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:13.626387 16221 tablet_bootstrap.cc:492] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:13.632535 16221 tablet_bootstrap.cc:654] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:13.633987 16221 log.cc:824] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:13.637005 16221 tablet_bootstrap.cc:492] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:13.637298 16221 ts_tablet_manager.cc:1367] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240501 08:12:13.644254 16221 raft_consensus.cc:348] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:13.644649 16221 raft_consensus.cc:374] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:13.644770 16221 raft_consensus.cc:729] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:13.645534 16221 consensus_queue.cc:260] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:13.647315 16221 ts_tablet_manager.cc:1398] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
W20240501 08:12:13.755929 15802 tablet.cc:2369] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:13.786118 16074 raft_consensus.cc:482] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:13.786293 16074 raft_consensus.cc:504] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:13.787046 16074 leader_election.cc:288] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
W20240501 08:12:13.792577 13411 tablet.cc:2369] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:13.793287 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6757fba154ce4915f4d447970ffd4" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:13.793560 13363 raft_consensus.cc:2454] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:13.794097 13040 leader_election.cc:302] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:13.794427 16074 raft_consensus.cc:2781] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:13.794528 16074 raft_consensus.cc:482] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:13.794598 16074 raft_consensus.cc:3037] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.797935 16074 raft_consensus.cc:504] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:13.798806 16074 leader_election.cc:288] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:13.799218 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6757fba154ce4915f4d447970ffd4" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:13.799383 13363 raft_consensus.cc:3037] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.802222 13363 raft_consensus.cc:2454] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:13.802635 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6757fba154ce4915f4d447970ffd4" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:13.802852 13040 leader_election.cc:302] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:13.802662 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e6757fba154ce4915f4d447970ffd4" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:13.803238 15756 raft_consensus.cc:2454] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:13.803521 15652 raft_consensus.cc:2781] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:13.803668 15652 raft_consensus.cc:686] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:13.804001 15652 consensus_queue.cc:237] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:13.807906 12830 catalog_manager.cc:5484] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
W20240501 08:12:13.853643 13151 tablet.cc:2369] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:13.924129 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:13.932824 13363 raft_consensus.cc:1261] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:13.933130 15755 raft_consensus.cc:3037] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:13.933521 16074 consensus_queue.cc:1035] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:12:13.936362 15755 raft_consensus.cc:1261] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:13.937999 15652 consensus_queue.cc:1035] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:13.943682 12830 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet b3e6757fba154ce4915f4d447970ffd4 (table rebalancer_dynamic_table [id=cd07cc53a527402f9e3860fb5443bda2]). Expected version 1 got 0
I20240501 08:12:13.946903 16233 tablet.cc:1722] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:13.949039 16233 tablet.cc:1620] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:13.962064 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=cd07cc53a527402f9e3860fb5443bda2] alter complete (version 1)
I20240501 08:12:13.965282 12832 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet b3e6757fba154ce4915f4d447970ffd4 (table rebalancer_dynamic_table [id=cd07cc53a527402f9e3860fb5443bda2]). Expected version 1 got 0
I20240501 08:12:13.969942 12830 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet b3e6757fba154ce4915f4d447970ffd4 (table rebalancer_dynamic_table [id=cd07cc53a527402f9e3860fb5443bda2]). Expected version 1 got 0
I20240501 08:12:13.971819 16235 tablet.cc:1722] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:13.973223 16235 tablet.cc:1620] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:13.983305 16230 mvcc.cc:205] Tried to move back new op lower bound from 7022801444581838848 to 7022801444063920128. Current Snapshot: MvccSnapshot[applied={T|T < 7022801444581838848}]
I20240501 08:12:13.988622 16234 tablet.cc:1722] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:13.990067 16234 tablet.cc:1620] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:13.991468 12830 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet b3e6757fba154ce4915f4d447970ffd4 (table rebalancer_dynamic_table [id=cd07cc53a527402f9e3860fb5443bda2]). Expected version 1 got 0
I20240501 08:12:14.028043 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:14.028730 12832 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 08:12:14.028959 12832 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240501 08:12:14.036438 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet e89f532aaa2f4e8e8155c1fc9b4767cc
I20240501 08:12:14.037253 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet ac31ed51cbd2474cb93c5393358848ec
I20240501 08:12:14.037689 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet e89f532aaa2f4e8e8155c1fc9b4767cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:14.038177 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet e89f532aaa2f4e8e8155c1fc9b4767cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:14.038693 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet e89f532aaa2f4e8e8155c1fc9b4767cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:14.040899 16070 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
I20240501 08:12:14.040920 16249 tablet_replica.cc:330] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:14.041518 16249 raft_consensus.cc:2229] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:14.042390 16249 raft_consensus.cc:2258] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.044517 16249 ts_tablet_manager.cc:1875] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.048107 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet ac31ed51cbd2474cb93c5393358848ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:14.049201 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet ac31ed51cbd2474cb93c5393358848ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:14.049300 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet baa03907546a4297a8eba0710a6b7acc
I20240501 08:12:14.049494 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet ac31ed51cbd2474cb93c5393358848ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:14.051019 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet baa03907546a4297a8eba0710a6b7acc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:14.051298 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet baa03907546a4297a8eba0710a6b7acc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:14.051846 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet baa03907546a4297a8eba0710a6b7acc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:14.052495 16249 ts_tablet_manager.cc:1888] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.052696 16249 log.cc:1196] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/e89f532aaa2f4e8e8155c1fc9b4767cc
I20240501 08:12:14.053316 16249 ts_tablet_manager.cc:1909] T e89f532aaa2f4e8e8155c1fc9b4767cc P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:14.055382 16249 tablet_replica.cc:330] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:14.055445 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet e89f532aaa2f4e8e8155c1fc9b4767cc (table rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]) successfully deleted
I20240501 08:12:14.055696 16249 raft_consensus.cc:2229] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.055970 16249 raft_consensus.cc:2258] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.057240 16249 ts_tablet_manager.cc:1875] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.062005 16250 tablet_replica.cc:330] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:14.062973 16248 tablet_replica.cc:330] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:14.063424 16248 raft_consensus.cc:2229] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.063792 16248 raft_consensus.cc:2258] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.064514 16250 raft_consensus.cc:2229] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.064855 16250 raft_consensus.cc:2258] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.066102 16249 ts_tablet_manager.cc:1888] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.066270 16249 log.cc:1196] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/baa03907546a4297a8eba0710a6b7acc
I20240501 08:12:14.066322 16248 ts_tablet_manager.cc:1875] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.066726 16249 ts_tablet_manager.cc:1909] T baa03907546a4297a8eba0710a6b7acc P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:14.068435 16251 tablet_replica.cc:330] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:14.068397 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet baa03907546a4297a8eba0710a6b7acc (table rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]) successfully deleted
I20240501 08:12:14.069001 16251 raft_consensus.cc:2229] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:14.069492 16251 raft_consensus.cc:2258] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.070580 16251 ts_tablet_manager.cc:1875] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.072268 16250 ts_tablet_manager.cc:1875] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.079278 12832 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:14.080893 16250 ts_tablet_manager.cc:1888] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.081090 16250 log.cc:1196] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/e89f532aaa2f4e8e8155c1fc9b4767cc
I20240501 08:12:14.081125 16251 ts_tablet_manager.cc:1888] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.081323 16251 log.cc:1196] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/ac31ed51cbd2474cb93c5393358848ec
I20240501 08:12:14.081699 16250 ts_tablet_manager.cc:1909] T e89f532aaa2f4e8e8155c1fc9b4767cc P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:14.082027 16251 ts_tablet_manager.cc:1909] T ac31ed51cbd2474cb93c5393358848ec P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:14.083638 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet e89f532aaa2f4e8e8155c1fc9b4767cc (table rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]) successfully deleted
I20240501 08:12:14.084046 16250 tablet_replica.cc:330] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:14.084007 16248 ts_tablet_manager.cc:1888] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.084035 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet ac31ed51cbd2474cb93c5393358848ec (table rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]) successfully deleted
I20240501 08:12:14.084179 16248 log.cc:1196] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/e89f532aaa2f4e8e8155c1fc9b4767cc
I20240501 08:12:14.084375 16250 raft_consensus.cc:2229] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.084437 16251 tablet_replica.cc:330] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:14.084641 16250 raft_consensus.cc:2258] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.084676 16251 raft_consensus.cc:2229] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:14.084908 16248 ts_tablet_manager.cc:1909] T e89f532aaa2f4e8e8155c1fc9b4767cc P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:14.084995 16251 raft_consensus.cc:2258] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.085469 16250 ts_tablet_manager.cc:1875] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.085880 16251 ts_tablet_manager.cc:1875] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.086768 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet e89f532aaa2f4e8e8155c1fc9b4767cc (table rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]) successfully deleted
I20240501 08:12:14.087152 16248 tablet_replica.cc:330] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:14.087406 16248 raft_consensus.cc:2229] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.087661 16248 raft_consensus.cc:2258] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.088426 16248 ts_tablet_manager.cc:1875] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.093621 16250 ts_tablet_manager.cc:1888] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.093887 16250 log.cc:1196] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/ac31ed51cbd2474cb93c5393358848ec
I20240501 08:12:14.095554 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 106da82bab514cab94c32effacf35691 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.096064 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106da82bab514cab94c32effacf35691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.096006 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 106da82bab514cab94c32effacf35691 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.096485 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106da82bab514cab94c32effacf35691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.104854 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 3e62b4a817174374a1ffdf9f8b25fb44 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.104844 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 3e62b4a817174374a1ffdf9f8b25fb44 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.105347 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e62b4a817174374a1ffdf9f8b25fb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.105466 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e62b4a817174374a1ffdf9f8b25fb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.107012 16234 tablet.cc:1722] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:14.107501 16234 tablet.cc:1620] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:14.108444 16233 tablet.cc:1722] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:14.108918 16233 tablet.cc:1620] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:14.110172 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 8953bcf510a2413cbceb4924f651b771 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.110606 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8953bcf510a2413cbceb4924f651b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.113607 16248 ts_tablet_manager.cc:1888] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.113762 16248 log.cc:1196] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/ac31ed51cbd2474cb93c5393358848ec
I20240501 08:12:14.114180 16248 ts_tablet_manager.cc:1909] T ac31ed51cbd2474cb93c5393358848ec P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:14.115171 16248 tablet_replica.cc:330] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:14.115334 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet ac31ed51cbd2474cb93c5393358848ec (table rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]) successfully deleted
I20240501 08:12:14.115392 16248 raft_consensus.cc:2229] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.115639 16248 raft_consensus.cc:2258] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.118966 16235 tablet.cc:1722] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:14.119501 16235 tablet.cc:1620] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:14.123077 16257 tablet_bootstrap.cc:492] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:14.136873 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=cd07cc53a527402f9e3860fb5443bda2] alter complete (version 2)
I20240501 08:12:14.144379 16221 tablet_bootstrap.cc:492] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:14.146461 16221 tablet_bootstrap.cc:654] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.150833 16257 tablet_bootstrap.cc:654] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.153857 16248 ts_tablet_manager.cc:1875] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.156093 16250 ts_tablet_manager.cc:1909] T ac31ed51cbd2474cb93c5393358848ec P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:14.156781 16251 ts_tablet_manager.cc:1888] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.156940 16251 log.cc:1196] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/baa03907546a4297a8eba0710a6b7acc
I20240501 08:12:14.157379 16251 ts_tablet_manager.cc:1909] T baa03907546a4297a8eba0710a6b7acc P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:14.158273 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet ac31ed51cbd2474cb93c5393358848ec (table rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]) successfully deleted
I20240501 08:12:14.158567 16221 tablet_bootstrap.cc:492] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:14.158627 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet baa03907546a4297a8eba0710a6b7acc (table rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]) successfully deleted
I20240501 08:12:14.158748 16221 ts_tablet_manager.cc:1367] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.002s
I20240501 08:12:14.160174 16221 raft_consensus.cc:348] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.160411 16221 raft_consensus.cc:374] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.160493 16221 raft_consensus.cc:729] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.160856 16256 tablet_bootstrap.cc:492] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:14.160846 16221 consensus_queue.cc:260] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.163995 16221 ts_tablet_manager.cc:1398] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:14.165417 16257 tablet_bootstrap.cc:492] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:14.165653 16257 ts_tablet_manager.cc:1367] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240501 08:12:14.165859 16256 tablet_bootstrap.cc:654] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.166613 16221 tablet_bootstrap.cc:492] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:14.167906 16248 ts_tablet_manager.cc:1888] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.168020 16248 log.cc:1196] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/baa03907546a4297a8eba0710a6b7acc
I20240501 08:12:14.168262 16257 raft_consensus.cc:348] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:14.168522 16248 ts_tablet_manager.cc:1909] T baa03907546a4297a8eba0710a6b7acc P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:14.168628 16257 raft_consensus.cc:374] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.168769 16257 raft_consensus.cc:729] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.169147 16257 consensus_queue.cc:260] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:14.169256 16256 tablet_bootstrap.cc:492] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:14.169389 16256 ts_tablet_manager.cc:1367] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240501 08:12:14.169631 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet baa03907546a4297a8eba0710a6b7acc (table rebalance_test_table_0 [id=bef23102be284c4a974cf25decec3985]) successfully deleted
I20240501 08:12:14.169790 16257 ts_tablet_manager.cc:1398] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:14.170185 16256 raft_consensus.cc:348] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:14.170374 16256 raft_consensus.cc:374] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.170966 16256 raft_consensus.cc:729] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.171229 16256 consensus_queue.cc:260] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:14.171829 16256 ts_tablet_manager.cc:1398] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:14.172164 16256 tablet_bootstrap.cc:492] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:14.173810 16256 tablet_bootstrap.cc:654] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.175789 16221 tablet_bootstrap.cc:654] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.177316 16256 tablet_bootstrap.cc:492] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:14.177585 16256 ts_tablet_manager.cc:1367] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:12:14.179781 16256 raft_consensus.cc:348] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.180279 16256 raft_consensus.cc:374] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.180519 16256 raft_consensus.cc:729] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.181011 16256 consensus_queue.cc:260] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.181700 16256 ts_tablet_manager.cc:1398] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:14.185137 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 106da82bab514cab94c32effacf35691 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.186430 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 106da82bab514cab94c32effacf35691. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.193643 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 8953bcf510a2413cbceb4924f651b771 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.196010 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8953bcf510a2413cbceb4924f651b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.199450 15808 raft_consensus.cc:482] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.199630 15808 raft_consensus.cc:504] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:14.200553 15808 leader_election.cc:288] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:14.201627 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106da82bab514cab94c32effacf35691" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:14.201869 12972 raft_consensus.cc:2454] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:14.202479 13171 leader_election.cc:302] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:14.203145 15808 raft_consensus.cc:2781] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.203279 15808 raft_consensus.cc:482] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.203375 15808 raft_consensus.cc:3037] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.206568 15808 raft_consensus.cc:504] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:14.207243 15808 leader_election.cc:288] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:14.209354 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106da82bab514cab94c32effacf35691" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:14.209595 12974 raft_consensus.cc:3037] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.212617 16223 raft_consensus.cc:482] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.212807 16223 raft_consensus.cc:504] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.214857 16223 leader_election.cc:288] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:14.222908 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106da82bab514cab94c32effacf35691" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
W20240501 08:12:14.228464 13168 leader_election.cc:341] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:14.228797 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "106da82bab514cab94c32effacf35691" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
W20240501 08:12:14.229777 13168 leader_election.cc:341] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:14.239496 12974 raft_consensus.cc:2454] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:14.240154 13171 leader_election.cc:302] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters: 4d516916eeac468db514eeb928f06190
I20240501 08:12:14.240768 16224 raft_consensus.cc:2781] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.240918 16224 raft_consensus.cc:686] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:14.241137 12827 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:14.241228 16224 consensus_queue.cc:237] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:14.249845 16274 tablet_bootstrap.cc:492] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:14.253458 16233 tablet.cc:1722] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:14.254392 16234 tablet.cc:1722] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:14.254892 16235 tablet.cc:1722] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:14.261955 16274 tablet_bootstrap.cc:654] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.265386 16274 log.cc:824] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:14.270602 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=cd07cc53a527402f9e3860fb5443bda2] alter complete (version 3)
I20240501 08:12:14.272821 16221 tablet_bootstrap.cc:492] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:14.273151 16221 ts_tablet_manager.cc:1367] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.107s user 0.012s sys 0.003s
I20240501 08:12:14.274040 16221 raft_consensus.cc:348] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:14.274395 16221 raft_consensus.cc:374] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.274521 16221 raft_consensus.cc:729] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.274888 16221 consensus_queue.cc:260] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:14.275764 16221 ts_tablet_manager.cc:1398] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:14.276126 16274 tablet_bootstrap.cc:492] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:14.276468 16274 ts_tablet_manager.cc:1367] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.007s
I20240501 08:12:14.287360 16274 raft_consensus.cc:348] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:14.287907 16274 raft_consensus.cc:374] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.288020 16274 raft_consensus.cc:729] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.288944 16274 consensus_queue.cc:260] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:14.295492 12832 catalog_manager.cc:5484] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:14.306289 16274 ts_tablet_manager.cc:1398] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.030s user 0.010s sys 0.009s
W20240501 08:12:14.314446 15689 leader_election.cc:341] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Not found: Tablet not found: 3e62b4a817174374a1ffdf9f8b25fb44
I20240501 08:12:14.318286 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 8953bcf510a2413cbceb4924f651b771 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.319819 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8953bcf510a2413cbceb4924f651b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.320389 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 3e62b4a817174374a1ffdf9f8b25fb44 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:14.320773 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e62b4a817174374a1ffdf9f8b25fb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.329717 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e62b4a817174374a1ffdf9f8b25fb44" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:14.329973 12972 raft_consensus.cc:2454] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:14.330596 15692 leader_election.cc:302] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters: 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:14.331148 16223 raft_consensus.cc:2781] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.331290 16223 raft_consensus.cc:482] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.331398 16223 raft_consensus.cc:3037] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.334880 16223 raft_consensus.cc:504] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.335875 16223 leader_election.cc:288] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
W20240501 08:12:14.337033 15689 leader_election.cc:341] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Not found: Tablet not found: 3e62b4a817174374a1ffdf9f8b25fb44
I20240501 08:12:14.338572 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e62b4a817174374a1ffdf9f8b25fb44" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:14.338824 12972 raft_consensus.cc:3037] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.341974 12972 raft_consensus.cc:2454] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
W20240501 08:12:14.342363 13281 tablet.cc:2369] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:14.346160 15692 leader_election.cc:302] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters: 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:14.346639 16223 raft_consensus.cc:2781] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.353971 16223 raft_consensus.cc:686] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:14.354948 16223 consensus_queue.cc:237] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.355568 16279 tablet_bootstrap.cc:492] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:14.368844 16267 tablet_bootstrap.cc:492] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:14.368634 12832 catalog_manager.cc:5484] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:14.369397 16279 tablet_bootstrap.cc:654] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.370926 16279 log.cc:824] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:14.373876 12832 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:14.374119 12832 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:14.375744 16267 tablet_bootstrap.cc:654] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.377760 16267 log.cc:824] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190: Log is configured to *not* fsync() on all Append() calls
I20240501 08:12:14.380441 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet b3e6757fba154ce4915f4d447970ffd4
W20240501 08:12:14.380945 13020 tablet.cc:2369] T 106da82bab514cab94c32effacf35691 P 76d830e199eb461c8ca689c9460ee0c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:14.381867 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e6757fba154ce4915f4d447970ffd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:14.382154 16250 tablet_replica.cc:330] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:14.382376 16250 raft_consensus.cc:2229] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:14.382908 16250 raft_consensus.cc:2258] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.383622 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e6757fba154ce4915f4d447970ffd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:14.383730 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e6757fba154ce4915f4d447970ffd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:14.384039 16251 tablet_replica.cc:330] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:14.384133 16250 ts_tablet_manager.cc:1875] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.384316 16251 raft_consensus.cc:2229] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.384586 16251 raft_consensus.cc:2258] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.389542 16267 tablet_bootstrap.cc:492] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:14.389880 16267 ts_tablet_manager.cc:1367] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.009s
I20240501 08:12:14.394342 16251 ts_tablet_manager.cc:1875] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.395430 16250 ts_tablet_manager.cc:1888] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:14.395555 16250 log.cc:1196] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/b3e6757fba154ce4915f4d447970ffd4
I20240501 08:12:14.396728 16267 raft_consensus.cc:348] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:14.397150 16267 raft_consensus.cc:374] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.397251 16267 raft_consensus.cc:729] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.397352 16250 ts_tablet_manager.cc:1909] T b3e6757fba154ce4915f4d447970ffd4 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:14.398090 16267 consensus_queue.cc:260] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:14.399966 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet b3e6757fba154ce4915f4d447970ffd4 (table rebalancer_dynamic_table_new_name [id=cd07cc53a527402f9e3860fb5443bda2]) successfully deleted
I20240501 08:12:14.406471 16267 ts_tablet_manager.cc:1398] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20240501 08:12:14.412765 16285 tablet_replica.cc:330] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:14.413195 16285 raft_consensus.cc:2229] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.413487 16285 raft_consensus.cc:2258] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.414958 16285 ts_tablet_manager.cc:1875] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.416574 16251 ts_tablet_manager.cc:1888] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:14.416769 16251 log.cc:1196] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/b3e6757fba154ce4915f4d447970ffd4
I20240501 08:12:14.419164 16251 ts_tablet_manager.cc:1909] T b3e6757fba154ce4915f4d447970ffd4 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:14.420392 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet b3e6757fba154ce4915f4d447970ffd4 (table rebalancer_dynamic_table_new_name [id=cd07cc53a527402f9e3860fb5443bda2]) successfully deleted
I20240501 08:12:14.422140 16279 tablet_bootstrap.cc:492] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:14.422518 16279 ts_tablet_manager.cc:1367] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.006s
I20240501 08:12:14.423530 16285 ts_tablet_manager.cc:1888] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:14.423712 16285 log.cc:1196] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/b3e6757fba154ce4915f4d447970ffd4
I20240501 08:12:14.424314 16285 ts_tablet_manager.cc:1909] T b3e6757fba154ce4915f4d447970ffd4 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:14.425993 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet b3e6757fba154ce4915f4d447970ffd4 (table rebalancer_dynamic_table_new_name [id=cd07cc53a527402f9e3860fb5443bda2]) successfully deleted
I20240501 08:12:14.435305 16279 raft_consensus.cc:348] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:14.435837 16279 raft_consensus.cc:374] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.435961 16279 raft_consensus.cc:729] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.437019 16279 consensus_queue.cc:260] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:14.438380 16203 heartbeater.cc:503] Master 127.6.56.126:44861 was elected leader, sending a full tablet report...
I20240501 08:12:14.439404 16279 ts_tablet_manager.cc:1398] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20240501 08:12:14.439911 16279 tablet_bootstrap.cc:492] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:14.442889 16279 tablet_bootstrap.cc:654] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.448348 16279 tablet_bootstrap.cc:492] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:14.448524 16279 ts_tablet_manager.cc:1367] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240501 08:12:14.449505 16279 raft_consensus.cc:348] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.449698 16279 raft_consensus.cc:374] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.449791 16279 raft_consensus.cc:729] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.450037 16279 consensus_queue.cc:260] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.450603 16279 ts_tablet_manager.cc:1398] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:14.461477 16289 raft_consensus.cc:482] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.461755 16289 raft_consensus.cc:504] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:14.463868 16289 leader_election.cc:288] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:14.482193 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e62b4a817174374a1ffdf9f8b25fb44" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:14.482553 12972 raft_consensus.cc:2379] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cc32f6e2d49477f9a23ce6e444039fb in current term 1: Already voted for candidate 982f49fc87f149439cda0de2261b6bfd in this term.
I20240501 08:12:14.482964 12832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:14.491598 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e62b4a817174374a1ffdf9f8b25fb44" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:14.492667 16092 leader_election.cc:302] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:14.492969 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 76c70be28cc94c99b43f5882fe90b028 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1865069970554a20a8dbf0b096a474ab]), partition=
I20240501 08:12:14.492936 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 76c70be28cc94c99b43f5882fe90b028 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1865069970554a20a8dbf0b096a474ab]), partition=
I20240501 08:12:14.493333 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c70be28cc94c99b43f5882fe90b028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.493340 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c70be28cc94c99b43f5882fe90b028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.493748 16289 raft_consensus.cc:3037] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.497887 16289 raft_consensus.cc:2726] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:14.499831 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 76c70be28cc94c99b43f5882fe90b028 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1865069970554a20a8dbf0b096a474ab]), partition=
I20240501 08:12:14.500355 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c70be28cc94c99b43f5882fe90b028. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:14.502717 16274 tablet_bootstrap.cc:492] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:14.504365 16274 tablet_bootstrap.cc:654] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.506649 16279 tablet_bootstrap.cc:492] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:14.508628 16279 tablet_bootstrap.cc:654] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:14.510224 16267 tablet_bootstrap.cc:492] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:14.512375 16267 tablet_bootstrap.cc:654] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
W20240501 08:12:14.518339 15939 tablet.cc:2369] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 08:12:14.520756 16071 tablet.cc:2369] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:14.521914 16279 tablet_bootstrap.cc:492] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:14.522092 16279 ts_tablet_manager.cc:1367] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20240501 08:12:14.523238 16279 raft_consensus.cc:348] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.523486 16279 raft_consensus.cc:374] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.523573 16279 raft_consensus.cc:729] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.523861 16279 consensus_queue.cc:260] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.525933 16279 ts_tablet_manager.cc:1398] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
W20240501 08:12:14.533707 16204 tablet.cc:2369] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:12:14.543612 16267 tablet_bootstrap.cc:492] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:14.543838 16267 ts_tablet_manager.cc:1367] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.001s
I20240501 08:12:14.544885 16274 tablet_bootstrap.cc:492] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:14.545120 16274 ts_tablet_manager.cc:1367] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.014s
I20240501 08:12:14.545017 16267 raft_consensus.cc:348] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.545322 16267 raft_consensus.cc:374] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.545480 16267 raft_consensus.cc:729] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.546312 16267 consensus_queue.cc:260] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.546360 16274 raft_consensus.cc:348] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.546730 16274 raft_consensus.cc:374] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:14.546870 16274 raft_consensus.cc:729] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:14.547098 16267 ts_tablet_manager.cc:1398] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:14.547237 16274 consensus_queue.cc:260] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.553431 12827 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 08:12:14.553790 12827 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240501 08:12:14.554760 16274 ts_tablet_manager.cc:1398] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 08:12:14.562978 12827 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet d968e8c981b041bbb290058b86506636
I20240501 08:12:14.563532 12827 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 2877095508b5400e9452adf3f737179a
I20240501 08:12:14.564111 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet d968e8c981b041bbb290058b86506636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:14.564157 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet d968e8c981b041bbb290058b86506636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:14.564194 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet d968e8c981b041bbb290058b86506636 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:14.564471 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet 2877095508b5400e9452adf3f737179a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:14.564508 16251 tablet_replica.cc:330] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:14.564731 12827 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 442304f2fddc4a80928a3b8a44ec17c6
I20240501 08:12:14.565593 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet 442304f2fddc4a80928a3b8a44ec17c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:14.568172 16248 tablet_replica.cc:330] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:14.568212 16251 raft_consensus.cc:2229] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.568468 16248 raft_consensus.cc:2229] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.568512 16251 raft_consensus.cc:2258] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.568691 16248 raft_consensus.cc:2258] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.569308 16251 ts_tablet_manager.cc:1875] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.569348 16248 ts_tablet_manager.cc:1875] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.572463 16250 tablet_replica.cc:330] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:14.572755 16250 raft_consensus.cc:2229] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:14.573100 16250 raft_consensus.cc:2258] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.573920 16250 ts_tablet_manager.cc:1875] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.576136 16251 ts_tablet_manager.cc:1888] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.576280 16251 log.cc:1196] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/d968e8c981b041bbb290058b86506636
I20240501 08:12:14.576915 16251 ts_tablet_manager.cc:1909] T d968e8c981b041bbb290058b86506636 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:14.577517 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet 2877095508b5400e9452adf3f737179a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:14.577570 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 2877095508b5400e9452adf3f737179a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:14.577845 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 442304f2fddc4a80928a3b8a44ec17c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:14.577950 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 442304f2fddc4a80928a3b8a44ec17c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:14 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:14.584506 16251 tablet_replica.cc:330] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:14.584687 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet d968e8c981b041bbb290058b86506636 (table rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]) successfully deleted
I20240501 08:12:14.585407 16250 ts_tablet_manager.cc:1888] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.585670 16250 log.cc:1196] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/d968e8c981b041bbb290058b86506636
I20240501 08:12:14.585760 16251 raft_consensus.cc:2229] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.586259 16250 ts_tablet_manager.cc:1909] T d968e8c981b041bbb290058b86506636 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:14.586586 16251 raft_consensus.cc:2258] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.587538 16251 ts_tablet_manager.cc:1875] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.587500 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet d968e8c981b041bbb290058b86506636 (table rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]) successfully deleted
I20240501 08:12:14.587790 16248 ts_tablet_manager.cc:1888] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.587977 16250 tablet_replica.cc:330] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:14.587991 16248 log.cc:1196] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/d968e8c981b041bbb290058b86506636
I20240501 08:12:14.588464 16250 raft_consensus.cc:2229] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:14.588557 16248 ts_tablet_manager.cc:1909] T d968e8c981b041bbb290058b86506636 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:14.588871 16250 raft_consensus.cc:2258] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.589933 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet d968e8c981b041bbb290058b86506636 (table rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]) successfully deleted
I20240501 08:12:14.590443 16250 ts_tablet_manager.cc:1875] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.592705 16248 tablet_replica.cc:330] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:14.593148 16248 raft_consensus.cc:2229] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.594739 16248 raft_consensus.cc:2258] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.595703 16248 ts_tablet_manager.cc:1875] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.595796 16306 tablet_replica.cc:330] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:14.600227 16251 ts_tablet_manager.cc:1888] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.600617 16306 raft_consensus.cc:2229] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:14.601097 16306 raft_consensus.cc:2258] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.602038 16306 ts_tablet_manager.cc:1875] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.607620 16250 ts_tablet_manager.cc:1888] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.608675 16306 ts_tablet_manager.cc:1888] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.608978 16251 log.cc:1196] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/2877095508b5400e9452adf3f737179a
I20240501 08:12:14.609233 16250 log.cc:1196] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/442304f2fddc4a80928a3b8a44ec17c6
I20240501 08:12:14.609565 16251 ts_tablet_manager.cc:1909] T 2877095508b5400e9452adf3f737179a P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:14.609798 16250 ts_tablet_manager.cc:1909] T 442304f2fddc4a80928a3b8a44ec17c6 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:14.610653 16251 tablet_replica.cc:330] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:14.610877 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 442304f2fddc4a80928a3b8a44ec17c6 (table rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]) successfully deleted
I20240501 08:12:14.610961 16251 raft_consensus.cc:2229] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.611207 16251 raft_consensus.cc:2258] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.611219 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 2877095508b5400e9452adf3f737179a (table rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]) successfully deleted
I20240501 08:12:14.611869 16251 ts_tablet_manager.cc:1875] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.616961 16251 ts_tablet_manager.cc:1888] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.617611 16306 log.cc:1196] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/2877095508b5400e9452adf3f737179a
I20240501 08:12:14.618144 16306 ts_tablet_manager.cc:1909] T 2877095508b5400e9452adf3f737179a P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:14.618264 16248 ts_tablet_manager.cc:1888] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.618434 16248 log.cc:1196] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/2877095508b5400e9452adf3f737179a
I20240501 08:12:14.619060 16248 ts_tablet_manager.cc:1909] T 2877095508b5400e9452adf3f737179a P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:14.619347 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 2877095508b5400e9452adf3f737179a (table rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]) successfully deleted
I20240501 08:12:14.620066 16251 log.cc:1196] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/442304f2fddc4a80928a3b8a44ec17c6
I20240501 08:12:14.620589 16251 ts_tablet_manager.cc:1909] T 442304f2fddc4a80928a3b8a44ec17c6 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:14.621757 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 442304f2fddc4a80928a3b8a44ec17c6 (table rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]) successfully deleted
I20240501 08:12:14.624747 16248 tablet_replica.cc:330] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:14.624738 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 2877095508b5400e9452adf3f737179a (table rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]) successfully deleted
I20240501 08:12:14.624992 16248 raft_consensus.cc:2229] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:14.625252 16248 raft_consensus.cc:2258] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:14.625977 16248 ts_tablet_manager.cc:1875] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:14.630887 16248 ts_tablet_manager.cc:1888] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:14.631059 16248 log.cc:1196] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/442304f2fddc4a80928a3b8a44ec17c6
I20240501 08:12:14.631428 16248 ts_tablet_manager.cc:1909] T 442304f2fddc4a80928a3b8a44ec17c6 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:14.632450 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 442304f2fddc4a80928a3b8a44ec17c6 (table rebalance_test_table_2 [id=73cd5808e32a4f1789c9e2a23bd16de2]) successfully deleted
I20240501 08:12:14.644806 16224 consensus_queue.cc:1035] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.657781 15892 raft_consensus.cc:3037] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.661602 15806 consensus_queue.cc:1035] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.689630 16276 raft_consensus.cc:482] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.689891 16276 raft_consensus.cc:504] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:14.691499 16276 leader_election.cc:288] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:14.698364 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953bcf510a2413cbceb4924f651b771" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:14.698640 15755 raft_consensus.cc:2454] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:14.700014 15959 leader_election.cc:302] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:14.700470 16276 raft_consensus.cc:2781] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.700580 16276 raft_consensus.cc:482] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.700641 16276 raft_consensus.cc:3037] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.700529 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953bcf510a2413cbceb4924f651b771" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:14.700934 16158 raft_consensus.cc:2454] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:14.703119 16276 raft_consensus.cc:504] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:14.703863 16276 leader_election.cc:288] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:14.704576 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953bcf510a2413cbceb4924f651b771" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:14.704772 16158 raft_consensus.cc:3037] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.705022 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8953bcf510a2413cbceb4924f651b771" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:14.705238 15755 raft_consensus.cc:3037] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.707765 16158 raft_consensus.cc:2454] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:14.708359 15957 leader_election.cc:302] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:14.708539 15755 raft_consensus.cc:2454] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:14.708714 16276 raft_consensus.cc:2781] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.709353 16276 raft_consensus.cc:686] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:14.714941 16276 consensus_queue.cc:237] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:14.720990 12830 catalog_manager.cc:5484] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:14.812507 16276 raft_consensus.cc:482] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.812750 16276 raft_consensus.cc:504] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.813580 16276 leader_election.cc:288] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:14.814311 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c70be28cc94c99b43f5882fe90b028" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:14.814574 16158 raft_consensus.cc:2454] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:14.815294 15957 leader_election.cc:302] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:14.815678 16276 raft_consensus.cc:2781] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:14.815788 16276 raft_consensus.cc:482] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:14.815901 16276 raft_consensus.cc:3037] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.818719 16276 raft_consensus.cc:504] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.819857 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c70be28cc94c99b43f5882fe90b028" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:14.820122 16158 raft_consensus.cc:3037] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.822383 16286 raft_consensus.cc:482] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:14.822556 16286 raft_consensus.cc:504] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.822860 16158 raft_consensus.cc:2454] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:14.823405 15957 leader_election.cc:302] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:14.823846 16334 raft_consensus.cc:2781] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:14.824139 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c70be28cc94c99b43f5882fe90b028" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:14.824426 15892 raft_consensus.cc:2454] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:14.824532 16286 leader_election.cc:288] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:14.826867 16276 leader_election.cc:288] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:14.832433 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c70be28cc94c99b43f5882fe90b028" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:14.832688 15892 raft_consensus.cc:3037] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:14.835801 15892 raft_consensus.cc:2454] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:14.837721 16334 raft_consensus.cc:686] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:14.838115 16334 consensus_queue.cc:237] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:14.852082 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c70be28cc94c99b43f5882fe90b028" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:14.852488 16158 raft_consensus.cc:2379] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d516916eeac468db514eeb928f06190 in current term 1: Already voted for candidate 5dd722c770d34d6c9154a87917117161 in this term.
I20240501 08:12:14.858331 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c70be28cc94c99b43f5882fe90b028" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:14.859541 15825 leader_election.cc:302] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:14.859897 16286 raft_consensus.cc:2726] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:14.862160 12826 catalog_manager.cc:5484] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:14.876389 16223 consensus_queue.cc:1035] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.880168 16372 consensus_queue.cc:1035] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:14.895577 15872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:14.951100 16005 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:14.959693 13344 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:14.977650 12832 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:14.995625 16137 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:14.999274 15892 raft_consensus.cc:1261] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:15.000262 16334 consensus_queue.cc:1035] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.020860 16158 raft_consensus.cc:1261] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:15.021898 16334 consensus_queue.cc:1035] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.047116 12832 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 76c70be28cc94c99b43f5882fe90b028 (table rebalancer_dynamic_table [id=1865069970554a20a8dbf0b096a474ab]). Expected version 1 got 0
I20240501 08:12:15.048036 13214 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:15.053671 16325 mvcc.cc:205] Tried to move back new op lower bound from 7022801448948600832 to 7022801448300044288. Current Snapshot: MvccSnapshot[applied={T|T < 7022801448948600832}]
I20240501 08:12:15.063320 16390 mvcc.cc:205] Tried to move back new op lower bound from 7022801448948600832 to 7022801448300044288. Current Snapshot: MvccSnapshot[applied={T|T < 7022801448948600832}]
I20240501 08:12:15.064882 15736 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:15.068034 12832 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 08:12:15.068313 12832 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240501 08:12:15.074954 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 67d42d76dff948b98a373831b515c7f7
I20240501 08:12:15.075464 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet ecda2d4abaac4ed6a31c5acff14177f3
I20240501 08:12:15.075762 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet ca8689eeb65744db8404fffcc84517b0
I20240501 08:12:15.077147 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet 67d42d76dff948b98a373831b515c7f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:15.077461 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 67d42d76dff948b98a373831b515c7f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:15.077598 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet ecda2d4abaac4ed6a31c5acff14177f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:15.077764 16306 tablet_replica.cc:330] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:15.077757 16248 tablet_replica.cc:330] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:15.077989 16306 raft_consensus.cc:2229] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.077987 16248 raft_consensus.cc:2229] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.078262 16306 raft_consensus.cc:2258] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.078264 16248 raft_consensus.cc:2258] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.078763 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 67d42d76dff948b98a373831b515c7f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:15.079000 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet ecda2d4abaac4ed6a31c5acff14177f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:15.079048 16248 ts_tablet_manager.cc:1875] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.079082 16306 ts_tablet_manager.cc:1875] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.079222 16250 tablet_replica.cc:330] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:15.079421 16250 raft_consensus.cc:2229] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:15.079727 16250 raft_consensus.cc:2258] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.080507 16250 ts_tablet_manager.cc:1875] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.077147 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8689eeb65744db8404fffcc84517b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:15.078769 13083 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8689eeb65744db8404fffcc84517b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:15.086213 16248 ts_tablet_manager.cc:1888] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.086386 16248 log.cc:1196] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/67d42d76dff948b98a373831b515c7f7
I20240501 08:12:15.086673 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet ecda2d4abaac4ed6a31c5acff14177f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:15.086949 16248 ts_tablet_manager.cc:1909] T 67d42d76dff948b98a373831b515c7f7 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:15.086961 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet ca8689eeb65744db8404fffcc84517b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:15.086975 16251 tablet_replica.cc:330] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:15.087256 16251 raft_consensus.cc:2229] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.087512 16251 raft_consensus.cc:2258] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.088014 16248 tablet_replica.cc:330] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:15.088232 16251 ts_tablet_manager.cc:1875] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.088212 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 67d42d76dff948b98a373831b515c7f7 (table rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]) successfully deleted
I20240501 08:12:15.088934 16248 raft_consensus.cc:2229] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.089169 16248 raft_consensus.cc:2258] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.089913 16248 ts_tablet_manager.cc:1875] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.093840 16250 ts_tablet_manager.cc:1888] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.094002 16250 log.cc:1196] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/67d42d76dff948b98a373831b515c7f7
I20240501 08:12:15.094434 16250 ts_tablet_manager.cc:1909] T 67d42d76dff948b98a373831b515c7f7 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:15.096508 12832 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 76c70be28cc94c99b43f5882fe90b028 (table rebalancer_dynamic_table [id=1865069970554a20a8dbf0b096a474ab]). Expected version 1 got 0
I20240501 08:12:15.096917 16396 tablet.cc:1722] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:15.097568 12830 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet 76c70be28cc94c99b43f5882fe90b028 (table rebalancer_dynamic_table [id=1865069970554a20a8dbf0b096a474ab]). Expected version 1 got 0
I20240501 08:12:15.098963 16397 tablet.cc:1722] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:15.101552 16396 tablet.cc:1620] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:15.103402 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 67d42d76dff948b98a373831b515c7f7 (table rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]) successfully deleted
I20240501 08:12:15.105953 12954 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:12:15.106038 16251 ts_tablet_manager.cc:1888] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.106509 16250 tablet_replica.cc:330] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:15.106822 16250 raft_consensus.cc:2229] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:15.107378 16250 raft_consensus.cc:2258] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.109056 16250 ts_tablet_manager.cc:1875] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.110504 16248 ts_tablet_manager.cc:1888] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.110774 16248 log.cc:1196] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/ecda2d4abaac4ed6a31c5acff14177f3
I20240501 08:12:15.111956 16248 ts_tablet_manager.cc:1909] T ecda2d4abaac4ed6a31c5acff14177f3 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:15.113207 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet ecda2d4abaac4ed6a31c5acff14177f3 (table rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]) successfully deleted
I20240501 08:12:15.100236 16397 tablet.cc:1620] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:15.124773 16306 ts_tablet_manager.cc:1888] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.106177 16251 log.cc:1196] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/ecda2d4abaac4ed6a31c5acff14177f3
I20240501 08:12:15.128876 16251 ts_tablet_manager.cc:1909] T ecda2d4abaac4ed6a31c5acff14177f3 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:15.129104 16306 log.cc:1196] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/67d42d76dff948b98a373831b515c7f7
I20240501 08:12:15.129611 16306 ts_tablet_manager.cc:1909] T 67d42d76dff948b98a373831b515c7f7 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:15.129698 12830 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet 76c70be28cc94c99b43f5882fe90b028 (table rebalancer_dynamic_table [id=1865069970554a20a8dbf0b096a474ab]). Expected version 1 got 0
I20240501 08:12:15.130875 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet ecda2d4abaac4ed6a31c5acff14177f3 (table rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]) successfully deleted
I20240501 08:12:15.130789 16398 tablet.cc:1722] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:15.132393 16398 tablet.cc:1620] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:15.135145 16306 tablet_replica.cc:330] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:15.136399 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 67d42d76dff948b98a373831b515c7f7 (table rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]) successfully deleted
I20240501 08:12:15.148034 13083 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:15.148221 12830 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=1865069970554a20a8dbf0b096a474ab] alter complete (version 1)
I20240501 08:12:15.156495 16251 tablet_replica.cc:330] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:15.156761 16251 raft_consensus.cc:2229] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.157044 16251 raft_consensus.cc:2258] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.158248 16251 ts_tablet_manager.cc:1875] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.162079 16306 raft_consensus.cc:2229] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.162357 16306 raft_consensus.cc:2258] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.163017 16306 ts_tablet_manager.cc:1875] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.167939 16251 ts_tablet_manager.cc:1888] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.168177 16251 log.cc:1196] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/ca8689eeb65744db8404fffcc84517b0
I20240501 08:12:15.168658 16250 ts_tablet_manager.cc:1888] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.168699 16251 ts_tablet_manager.cc:1909] T ca8689eeb65744db8404fffcc84517b0 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:15.168874 16250 log.cc:1196] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/ecda2d4abaac4ed6a31c5acff14177f3
I20240501 08:12:15.169835 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet ca8689eeb65744db8404fffcc84517b0 (table rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]) successfully deleted
I20240501 08:12:15.171651 16250 ts_tablet_manager.cc:1909] T ecda2d4abaac4ed6a31c5acff14177f3 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:15.171698 16306 ts_tablet_manager.cc:1888] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.172158 16306 log.cc:1196] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/ca8689eeb65744db8404fffcc84517b0
I20240501 08:12:15.172791 16250 tablet_replica.cc:330] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:15.172994 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet ecda2d4abaac4ed6a31c5acff14177f3 (table rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]) successfully deleted
I20240501 08:12:15.173121 16250 raft_consensus.cc:2229] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:15.173482 16250 raft_consensus.cc:2258] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.173647 16306 ts_tablet_manager.cc:1909] T ca8689eeb65744db8404fffcc84517b0 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:15.175189 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet ca8689eeb65744db8404fffcc84517b0 (table rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]) successfully deleted
I20240501 08:12:15.175781 16250 ts_tablet_manager.cc:1875] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.190191 16250 ts_tablet_manager.cc:1888] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.190480 16250 log.cc:1196] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/ca8689eeb65744db8404fffcc84517b0
I20240501 08:12:15.190992 16250 ts_tablet_manager.cc:1909] T ca8689eeb65744db8404fffcc84517b0 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:15.192270 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet ca8689eeb65744db8404fffcc84517b0 (table rebalance_test_table_4 [id=0248d0b90c4d46e2ba0b50131b32f5d8]) successfully deleted
I20240501 08:12:15.268498 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:15.279084 16396 tablet.cc:1722] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:15.279618 16396 tablet.cc:1620] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:15.280844 16397 tablet.cc:1722] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:15.281330 16397 tablet.cc:1620] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:15.285877 16398 tablet.cc:1722] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:15.286374 16398 tablet.cc:1620] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:15.289374 12830 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=1865069970554a20a8dbf0b096a474ab] alter complete (version 2)
I20240501 08:12:15.295125 16334 consensus_queue.cc:1035] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.309075 16334 consensus_queue.cc:1035] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:15.356124 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:15.367501 15736 tablet_service.cc:1467] Processing CreateTablet for tablet bdabd94a46dc4a5796949fdf9d0c698c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.367765 16005 tablet_service.cc:1467] Processing CreateTablet for tablet bdabd94a46dc4a5796949fdf9d0c698c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.368013 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdabd94a46dc4a5796949fdf9d0c698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.368165 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdabd94a46dc4a5796949fdf9d0c698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.376139 15872 tablet_service.cc:1467] Processing CreateTablet for tablet bdabd94a46dc4a5796949fdf9d0c698c (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.376339 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 85056837a567420a8b480b34b6017b40 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.376684 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdabd94a46dc4a5796949fdf9d0c698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.376713 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85056837a567420a8b480b34b6017b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.384497 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 85056837a567420a8b480b34b6017b40 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.385056 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85056837a567420a8b480b34b6017b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.385574 13083 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2c8548c31343ff9426fe7eda194703 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.385967 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2c8548c31343ff9426fe7eda194703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.388684 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2c8548c31343ff9426fe7eda194703 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.389102 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2c8548c31343ff9426fe7eda194703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.393646 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 6c2c8548c31343ff9426fe7eda194703 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.394140 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2c8548c31343ff9426fe7eda194703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.396991 15871 tablet_service.cc:1467] Processing CreateTablet for tablet 85056837a567420a8b480b34b6017b40 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:15.397495 15871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85056837a567420a8b480b34b6017b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.416141 16438 tablet_bootstrap.cc:492] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:15.416980 16437 tablet_bootstrap.cc:492] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:15.417408 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:15.418340 16439 tablet_bootstrap.cc:492] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:15.419915 16437 tablet_bootstrap.cc:654] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.421471 16435 tablet_bootstrap.cc:492] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:15.424080 16435 tablet_bootstrap.cc:654] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.427206 16438 tablet_bootstrap.cc:654] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.427913 16440 tablet_bootstrap.cc:492] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:15.430199 16396 tablet.cc:1722] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:15.430393 16439 tablet_bootstrap.cc:654] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.432442 16440 tablet_bootstrap.cc:654] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.432763 16398 tablet.cc:1722] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:15.436968 16397 tablet.cc:1722] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:15.441833 16433 tablet_bootstrap.cc:492] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:15.444674 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=1865069970554a20a8dbf0b096a474ab] alter complete (version 3)
I20240501 08:12:15.449002 16433 tablet_bootstrap.cc:654] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.451048 16440 tablet_bootstrap.cc:492] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:15.451273 16440 ts_tablet_manager.cc:1367] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240501 08:12:15.452103 16440 raft_consensus.cc:348] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.452466 16440 raft_consensus.cc:374] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.452595 16440 raft_consensus.cc:729] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.452895 16440 consensus_queue.cc:260] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.453624 16440 ts_tablet_manager.cc:1398] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:15.454748 16435 tablet_bootstrap.cc:492] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:15.454969 16435 ts_tablet_manager.cc:1367] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.006s
I20240501 08:12:15.457373 16437 tablet_bootstrap.cc:492] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:15.457600 16437 ts_tablet_manager.cc:1367] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.000s
I20240501 08:12:15.458616 16437 raft_consensus.cc:348] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.458885 16437 raft_consensus.cc:374] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.459038 16437 raft_consensus.cc:729] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.459414 16437 consensus_queue.cc:260] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.459699 16439 tablet_bootstrap.cc:492] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:15.460186 16437 ts_tablet_manager.cc:1398] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:15.460611 16437 tablet_bootstrap.cc:492] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:15.460700 16439 ts_tablet_manager.cc:1367] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.004s
I20240501 08:12:15.456221 16435 raft_consensus.cc:348] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:15.463059 16438 tablet_bootstrap.cc:492] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:15.462487 16435 raft_consensus.cc:374] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.463676 16435 raft_consensus.cc:729] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.464025 16438 ts_tablet_manager.cc:1367] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.000s
I20240501 08:12:15.464004 16435 consensus_queue.cc:260] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:15.464512 16439 raft_consensus.cc:348] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:15.464670 16435 ts_tablet_manager.cc:1398] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:15.465085 16437 tablet_bootstrap.cc:654] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.465263 16439 raft_consensus.cc:374] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.465803 16438 raft_consensus.cc:348] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.466094 16439 raft_consensus.cc:729] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.468658 16439 consensus_queue.cc:260] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:15.469081 16438 raft_consensus.cc:374] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.469355 16438 raft_consensus.cc:729] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.474938 16439 ts_tablet_manager.cc:1398] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 08:12:15.475365 16438 consensus_queue.cc:260] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.476332 16438 ts_tablet_manager.cc:1398] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:12:15.485211 16433 tablet_bootstrap.cc:492] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:15.485494 16433 ts_tablet_manager.cc:1367] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.000s
I20240501 08:12:15.486831 16433 raft_consensus.cc:348] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.487154 16433 raft_consensus.cc:374] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.487299 16433 raft_consensus.cc:729] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.487699 16433 consensus_queue.cc:260] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.491036 16433 ts_tablet_manager.cc:1398] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:15.491385 16433 tablet_bootstrap.cc:492] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:15.492141 16437 tablet_bootstrap.cc:492] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:15.492364 16437 ts_tablet_manager.cc:1367] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.005s
I20240501 08:12:15.493202 16437 raft_consensus.cc:348] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.493463 16437 raft_consensus.cc:374] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.493549 16437 raft_consensus.cc:729] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.493831 16437 consensus_queue.cc:260] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.494004 16438 tablet_bootstrap.cc:492] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:15.495117 16437 ts_tablet_manager.cc:1398] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:15.497541 16438 tablet_bootstrap.cc:654] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.501130 16433 tablet_bootstrap.cc:654] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.511472 16433 tablet_bootstrap.cc:492] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:15.511772 16433 ts_tablet_manager.cc:1367] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240501 08:12:15.512598 16438 tablet_bootstrap.cc:492] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:15.512833 16438 ts_tablet_manager.cc:1367] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.004s
I20240501 08:12:15.512773 16433 raft_consensus.cc:348] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.513144 16433 raft_consensus.cc:374] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.513264 16433 raft_consensus.cc:729] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.513586 16433 consensus_queue.cc:260] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.516680 16433 ts_tablet_manager.cc:1398] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:15.513613 16438 raft_consensus.cc:348] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:15.531010 16438 raft_consensus.cc:374] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.531128 16438 raft_consensus.cc:729] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.531404 16438 consensus_queue.cc:260] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:15.532063 16438 ts_tablet_manager.cc:1398] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240501 08:12:15.563063 12832 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:15.563802 12832 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:15.568236 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 76c70be28cc94c99b43f5882fe90b028
I20240501 08:12:15.569245 16137 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c70be28cc94c99b43f5882fe90b028 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:15.574857 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c70be28cc94c99b43f5882fe90b028 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:15.575481 16458 tablet_replica.cc:330] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:15.575942 16458 raft_consensus.cc:2229] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.576349 16458 raft_consensus.cc:2258] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.578564 16458 ts_tablet_manager.cc:1875] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.578598 12828 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 08:12:15.578816 12828 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240501 08:12:15.584095 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 65842be1fe3c469dba1b57afdb5fb948
I20240501 08:12:15.584136 15871 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c70be28cc94c99b43f5882fe90b028 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:15.585239 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 65842be1fe3c469dba1b57afdb5fb948 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:15.585649 16248 tablet_replica.cc:330] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:15.585953 16248 raft_consensus.cc:2229] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.586297 16248 raft_consensus.cc:2258] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.587294 16248 ts_tablet_manager.cc:1875] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.588332 16459 tablet_replica.cc:330] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:15.588907 16459 raft_consensus.cc:2229] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:15.589635 16459 raft_consensus.cc:2258] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.591632 16459 ts_tablet_manager.cc:1875] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.594672 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 75a875e42dcb4b6e851dd7ab8589a03b
I20240501 08:12:15.595129 13083 tablet_service.cc:1514] Processing DeleteTablet for tablet 65842be1fe3c469dba1b57afdb5fb948 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:15.595373 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 22dc5fac8a454c85af08f20c30ff96ff
I20240501 08:12:15.595588 16250 tablet_replica.cc:330] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:15.595657 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a875e42dcb4b6e851dd7ab8589a03b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:15.595819 16250 raft_consensus.cc:2229] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:15.596199 16250 raft_consensus.cc:2258] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.596340 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 22dc5fac8a454c85af08f20c30ff96ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:15.596412 16289 raft_consensus.cc:482] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.596598 16289 raft_consensus.cc:504] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:15.599247 16459 ts_tablet_manager.cc:1888] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:15.599416 16459 log.cc:1196] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/76c70be28cc94c99b43f5882fe90b028
I20240501 08:12:15.599967 16459 ts_tablet_manager.cc:1909] T 76c70be28cc94c99b43f5882fe90b028 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:15.599469 16306 tablet_replica.cc:330] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:15.601858 16306 raft_consensus.cc:2229] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:15.602218 16306 raft_consensus.cc:2258] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.602563 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet 65842be1fe3c469dba1b57afdb5fb948 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:15.602866 16251 tablet_replica.cc:330] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:15.603017 16306 ts_tablet_manager.cc:1875] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.603062 16251 raft_consensus.cc:2229] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.603312 16251 raft_consensus.cc:2258] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.603981 16251 ts_tablet_manager.cc:1875] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.605377 16250 ts_tablet_manager.cc:1875] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.608369 16289 leader_election.cc:288] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:15.613978 16248 ts_tablet_manager.cc:1888] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.614181 16248 log.cc:1196] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/65842be1fe3c469dba1b57afdb5fb948
I20240501 08:12:15.614742 16248 ts_tablet_manager.cc:1909] T 65842be1fe3c469dba1b57afdb5fb948 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:15.616211 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 65842be1fe3c469dba1b57afdb5fb948 (table rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]) successfully deleted
I20240501 08:12:15.618567 16250 ts_tablet_manager.cc:1888] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.618712 16250 log.cc:1196] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/65842be1fe3c469dba1b57afdb5fb948
I20240501 08:12:15.619197 16250 ts_tablet_manager.cc:1909] T 65842be1fe3c469dba1b57afdb5fb948 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:15.623514 16306 ts_tablet_manager.cc:1888] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.623677 16306 log.cc:1196] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/75a875e42dcb4b6e851dd7ab8589a03b
I20240501 08:12:15.624136 16306 ts_tablet_manager.cc:1909] T 75a875e42dcb4b6e851dd7ab8589a03b P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:15.624425 13083 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a875e42dcb4b6e851dd7ab8589a03b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:15.624673 16248 tablet_replica.cc:330] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:15.625026 16248 raft_consensus.cc:2229] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:15.625638 16248 raft_consensus.cc:2258] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.625829 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 65842be1fe3c469dba1b57afdb5fb948 (table rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]) successfully deleted
I20240501 08:12:15.626092 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 76c70be28cc94c99b43f5882fe90b028 (table rebalancer_dynamic_table_new_name [id=1865069970554a20a8dbf0b096a474ab]) successfully deleted
I20240501 08:12:15.626652 16248 ts_tablet_manager.cc:1875] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.626688 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet 22dc5fac8a454c85af08f20c30ff96ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:15.626938 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a875e42dcb4b6e851dd7ab8589a03b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:15.628778 13083 tablet_service.cc:1514] Processing DeleteTablet for tablet 22dc5fac8a454c85af08f20c30ff96ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:15 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:15.629683 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 75a875e42dcb4b6e851dd7ab8589a03b (table rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]) successfully deleted
I20240501 08:12:15.635618 16458 ts_tablet_manager.cc:1888] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:15.635988 16458 log.cc:1196] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/76c70be28cc94c99b43f5882fe90b028
I20240501 08:12:15.636595 16463 tablet_replica.cc:330] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:15.636902 16458 ts_tablet_manager.cc:1909] T 76c70be28cc94c99b43f5882fe90b028 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:15.637154 16463 raft_consensus.cc:2229] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.637724 16463 raft_consensus.cc:2258] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.640252 16463 ts_tablet_manager.cc:1875] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.646315 16251 ts_tablet_manager.cc:1888] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.647850 16251 log.cc:1196] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/65842be1fe3c469dba1b57afdb5fb948
I20240501 08:12:15.648180 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 76c70be28cc94c99b43f5882fe90b028 (table rebalancer_dynamic_table_new_name [id=1865069970554a20a8dbf0b096a474ab]) successfully deleted
I20240501 08:12:15.649267 16251 ts_tablet_manager.cc:1909] T 65842be1fe3c469dba1b57afdb5fb948 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:15.651309 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 65842be1fe3c469dba1b57afdb5fb948 (table rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]) successfully deleted
I20240501 08:12:15.651605 16463 ts_tablet_manager.cc:1888] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:15.651772 16463 log.cc:1196] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/76c70be28cc94c99b43f5882fe90b028
I20240501 08:12:15.652098 16250 tablet_replica.cc:330] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:15.652566 16250 raft_consensus.cc:2229] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.653334 16248 ts_tablet_manager.cc:1888] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.653822 16463 ts_tablet_manager.cc:1909] T 76c70be28cc94c99b43f5882fe90b028 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:15.653985 16248 log.cc:1196] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/22dc5fac8a454c85af08f20c30ff96ff
I20240501 08:12:15.654495 16248 ts_tablet_manager.cc:1909] T 22dc5fac8a454c85af08f20c30ff96ff P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:15.655633 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 22dc5fac8a454c85af08f20c30ff96ff (table rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]) successfully deleted
I20240501 08:12:15.656064 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 76c70be28cc94c99b43f5882fe90b028 (table rebalancer_dynamic_table_new_name [id=1865069970554a20a8dbf0b096a474ab]) successfully deleted
I20240501 08:12:15.657167 16250 raft_consensus.cc:2258] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.657960 16251 tablet_replica.cc:330] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:15.658200 16251 raft_consensus.cc:2229] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.658435 16251 raft_consensus.cc:2258] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.659426 16250 ts_tablet_manager.cc:1875] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.664084 15871 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:12:15.665815 16251 ts_tablet_manager.cc:1875] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.672760 16251 ts_tablet_manager.cc:1888] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.672926 16251 log.cc:1196] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/22dc5fac8a454c85af08f20c30ff96ff
I20240501 08:12:15.673339 16251 ts_tablet_manager.cc:1909] T 22dc5fac8a454c85af08f20c30ff96ff P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:15.674300 16251 tablet_replica.cc:330] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:15.676481 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 22dc5fac8a454c85af08f20c30ff96ff (table rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]) successfully deleted
I20240501 08:12:15.677645 16250 ts_tablet_manager.cc:1888] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.677799 16250 log.cc:1196] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/75a875e42dcb4b6e851dd7ab8589a03b
I20240501 08:12:15.678229 16250 ts_tablet_manager.cc:1909] T 75a875e42dcb4b6e851dd7ab8589a03b P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:15.679227 16250 tablet_replica.cc:330] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:15.679446 16250 raft_consensus.cc:2229] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.679708 16250 raft_consensus.cc:2258] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.680269 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 75a875e42dcb4b6e851dd7ab8589a03b (table rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]) successfully deleted
I20240501 08:12:15.680457 16250 ts_tablet_manager.cc:1875] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.686787 12828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:15.687655 13345 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:15.689116 16250 ts_tablet_manager.cc:1888] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.689255 16250 log.cc:1196] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/22dc5fac8a454c85af08f20c30ff96ff
I20240501 08:12:15.689674 16250 ts_tablet_manager.cc:1909] T 22dc5fac8a454c85af08f20c30ff96ff P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:15.690768 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 22dc5fac8a454c85af08f20c30ff96ff (table rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]) successfully deleted
I20240501 08:12:15.693351 16251 raft_consensus.cc:2229] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:15.693655 16251 raft_consensus.cc:2258] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:15.694427 16251 ts_tablet_manager.cc:1875] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:15.705327 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2c8548c31343ff9426fe7eda194703" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:15.705591 13104 raft_consensus.cc:2454] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:15.706069 16092 leader_election.cc:302] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:15.706450 16289 raft_consensus.cc:2781] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:15.706558 16289 raft_consensus.cc:482] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:15.706653 16289 raft_consensus.cc:3037] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.706727 13345 tablet_service.cc:1467] Processing CreateTablet for tablet fbaf0a39a37a498b8f341cf1f643fa59 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16dfc2830de84c11b1903858a11263a3]), partition=
I20240501 08:12:15.707099 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbaf0a39a37a498b8f341cf1f643fa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.709501 16289 raft_consensus.cc:504] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:15.710490 16005 tablet_service.cc:1467] Processing CreateTablet for tablet fbaf0a39a37a498b8f341cf1f643fa59 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16dfc2830de84c11b1903858a11263a3]), partition=
I20240501 08:12:15.710847 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbaf0a39a37a498b8f341cf1f643fa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.711925 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2c8548c31343ff9426fe7eda194703" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:15.712369 13233 raft_consensus.cc:2454] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:15.710055 16137 tablet_service.cc:1467] Processing CreateTablet for tablet fbaf0a39a37a498b8f341cf1f643fa59 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=16dfc2830de84c11b1903858a11263a3]), partition=
I20240501 08:12:15.717615 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbaf0a39a37a498b8f341cf1f643fa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:15.717782 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2c8548c31343ff9426fe7eda194703" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:15.717998 13104 raft_consensus.cc:3037] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.719074 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2c8548c31343ff9426fe7eda194703" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:15.719332 13233 raft_consensus.cc:3037] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.722715 13104 raft_consensus.cc:2454] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:15.723132 16092 leader_election.cc:302] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:15.723158 13233 raft_consensus.cc:2454] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:15.723668 16289 leader_election.cc:288] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:15.723982 16289 raft_consensus.cc:482] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.724361 16289 raft_consensus.cc:504] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.725462 16289 leader_election.cc:288] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:15.726528 16289 raft_consensus.cc:2781] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:15.727301 16289 raft_consensus.cc:686] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:15.727982 16289 consensus_queue.cc:237] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:15.729658 16438 tablet_bootstrap.cc:492] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:15.731858 16438 tablet_bootstrap.cc:654] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.733465 16286 raft_consensus.cc:482] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.733652 16286 raft_consensus.cc:504] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.734647 16286 leader_election.cc:288] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:15.736322 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85056837a567420a8b480b34b6017b40" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:15.736608 16158 raft_consensus.cc:2454] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:15.737942 15823 leader_election.cc:302] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:15.740204 16286 raft_consensus.cc:2781] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:15.740353 16286 raft_consensus.cc:482] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:15.740456 16286 raft_consensus.cc:3037] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.742496 16440 tablet_bootstrap.cc:492] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:15.743710 16286 raft_consensus.cc:504] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.744215 16440 tablet_bootstrap.cc:654] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.751874 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85056837a567420a8b480b34b6017b40" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:15.752174 16158 raft_consensus.cc:3037] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.756453 16158 raft_consensus.cc:2454] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:15.757493 16372 raft_consensus.cc:482] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.757687 16372 raft_consensus.cc:504] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.758517 16372 leader_election.cc:288] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:15.758757 15823 leader_election.cc:302] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:15.759948 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85056837a567420a8b480b34b6017b40" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:15.760335 16157 raft_consensus.cc:2379] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 982f49fc87f149439cda0de2261b6bfd in current term 1: Already voted for candidate 4d516916eeac468db514eeb928f06190 in this term.
I20240501 08:12:15.788236 16286 leader_election.cc:288] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:15.788576 16286 raft_consensus.cc:2781] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:15.788744 16286 raft_consensus.cc:686] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:15.789254 16286 consensus_queue.cc:237] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.797396 16005 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:15.801401 12828 catalog_manager.cc:5484] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
W20240501 08:12:15.802322 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.66:35509, user_credentials={real_user=slave}} blocked reactor thread for 79210.8us
I20240501 08:12:15.803866 16438 tablet_bootstrap.cc:492] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:15.804066 16438 ts_tablet_manager.cc:1367] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.007s
I20240501 08:12:15.805017 16438 raft_consensus.cc:348] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:15.805262 16438 raft_consensus.cc:374] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.805359 16438 raft_consensus.cc:729] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.805550 16438 consensus_queue.cc:260] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:15.806360 16438 ts_tablet_manager.cc:1398] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:15.806880 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85056837a567420a8b480b34b6017b40" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:15.807142 15755 raft_consensus.cc:2454] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:15.829324 16251 ts_tablet_manager.cc:1888] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:15.831835 16251 log.cc:1196] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/75a875e42dcb4b6e851dd7ab8589a03b
I20240501 08:12:15.834280 16251 ts_tablet_manager.cc:1909] T 75a875e42dcb4b6e851dd7ab8589a03b P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:15.836076 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 75a875e42dcb4b6e851dd7ab8589a03b (table rebalance_test_table_6 [id=21a198e8f040480a9343868c04aa14a6]) successfully deleted
I20240501 08:12:15.840131 16092 leader_election.cc:302] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:15.844383 16473 raft_consensus.cc:2741] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240501 08:12:15.845363 16475 tablet_bootstrap.cc:492] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:15.855553 16475 tablet_bootstrap.cc:654] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:15.870726 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85056837a567420a8b480b34b6017b40" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
W20240501 08:12:15.870920 12909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 145595us
I20240501 08:12:15.882833 16440 tablet_bootstrap.cc:492] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:15.883158 16440 ts_tablet_manager.cc:1367] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.013s
I20240501 08:12:15.884089 16440 raft_consensus.cc:348] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:15.884613 16440 raft_consensus.cc:374] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.884752 16440 raft_consensus.cc:729] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.885066 16440 consensus_queue.cc:260] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:15.885439 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85056837a567420a8b480b34b6017b40" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:15.886029 16372 raft_consensus.cc:482] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.886211 16372 raft_consensus.cc:504] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.886476 16440 ts_tablet_manager.cc:1398] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20240501 08:12:15.884101 15823 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 125329us
I20240501 08:12:15.887051 16372 leader_election.cc:288] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:15.887642 15689 leader_election.cc:302] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:15.888254 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdabd94a46dc4a5796949fdf9d0c698c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:15.888062 12828 catalog_manager.cc:5484] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:15.888509 15893 raft_consensus.cc:2454] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:15.889901 16372 raft_consensus.cc:3037] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.893164 16372 raft_consensus.cc:2726] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:15.896651 16367 raft_consensus.cc:482] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:15.896823 16367 raft_consensus.cc:504] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.897580 16367 leader_election.cc:288] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:15.898458 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdabd94a46dc4a5796949fdf9d0c698c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:15.898694 15893 raft_consensus.cc:2454] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:15.899309 15957 leader_election.cc:302] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:15.899646 16334 raft_consensus.cc:2781] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:15.899755 16334 raft_consensus.cc:482] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:15.899835 16334 raft_consensus.cc:3037] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.902856 13215 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:15.902750 16334 raft_consensus.cc:504] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.903877 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdabd94a46dc4a5796949fdf9d0c698c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:15.904073 16334 leader_election.cc:288] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:15.904116 15893 raft_consensus.cc:3037] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.905030 15689 leader_election.cc:302] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:15.907100 15893 raft_consensus.cc:2454] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:15.907799 15957 leader_election.cc:302] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:15.908248 16334 raft_consensus.cc:2781] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:15.908399 16334 raft_consensus.cc:686] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:15.908676 16334 consensus_queue.cc:237] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.911329 16372 raft_consensus.cc:2781] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:15.911546 16372 raft_consensus.cc:482] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:15.911680 16372 raft_consensus.cc:3037] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:15.915158 16372 raft_consensus.cc:504] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:15.916165 16372 leader_election.cc:288] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:15.916625 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdabd94a46dc4a5796949fdf9d0c698c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:15.916920 15755 raft_consensus.cc:2379] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5dd722c770d34d6c9154a87917117161 in current term 1: Already voted for candidate 982f49fc87f149439cda0de2261b6bfd in this term.
I20240501 08:12:15.917827 13083 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:15.918053 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdabd94a46dc4a5796949fdf9d0c698c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:15.918632 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdabd94a46dc4a5796949fdf9d0c698c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:15.920207 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdabd94a46dc4a5796949fdf9d0c698c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:15.920517 15755 raft_consensus.cc:2379] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5dd722c770d34d6c9154a87917117161 in current term 1: Already voted for candidate 982f49fc87f149439cda0de2261b6bfd in this term.
I20240501 08:12:15.921317 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdabd94a46dc4a5796949fdf9d0c698c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:15.921697 15893 raft_consensus.cc:2379] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 982f49fc87f149439cda0de2261b6bfd in current term 1: Already voted for candidate 5dd722c770d34d6c9154a87917117161 in this term.
I20240501 08:12:15.927423 15689 leader_election.cc:302] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161
I20240501 08:12:15.928838 16475 tablet_bootstrap.cc:492] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:15.929065 16475 ts_tablet_manager.cc:1367] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.000s
I20240501 08:12:15.930470 16475 raft_consensus.cc:348] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:15.930788 16475 raft_consensus.cc:374] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:15.930999 16475 raft_consensus.cc:729] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:15.931684 16475 consensus_queue.cc:260] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:15.933260 16475 ts_tablet_manager.cc:1398] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:15.931203 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85056837a567420a8b480b34b6017b40" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:15.933524 15755 raft_consensus.cc:2454] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:15.931320 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85056837a567420a8b480b34b6017b40" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:15.937337 15756 raft_consensus.cc:2454] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:15.937500 16137 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:15.938014 16495 raft_consensus.cc:2726] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:12:15.956612 12828 catalog_manager.cc:5484] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:15.963454 12954 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:15.964098 15736 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:16.119949 16473 raft_consensus.cc:482] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.120203 16473 raft_consensus.cc:504] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.121201 16473 leader_election.cc:288] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:16.125510 12828 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 08:12:16.125763 12828 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240501 08:12:16.129240 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbaf0a39a37a498b8f341cf1f643fa59" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:16.129266 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbaf0a39a37a498b8f341cf1f643fa59" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:16.129501 13364 raft_consensus.cc:2454] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:16.129521 16024 raft_consensus.cc:2454] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:16.130110 16092 leader_election.cc:302] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:16.130527 16473 raft_consensus.cc:2781] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.130625 16473 raft_consensus.cc:482] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.130717 16473 raft_consensus.cc:3037] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.132243 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet b3fcc47f0a244a08b3adb00faa790da6
I20240501 08:12:16.132828 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 76bd7c097eef44b2a35d3adc581e0cdf
I20240501 08:12:16.133458 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet b3fcc47f0a244a08b3adb00faa790da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:16.133592 16473 raft_consensus.cc:504] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.133770 16251 tablet_replica.cc:330] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:16.134009 16251 raft_consensus.cc:2229] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.134152 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet 76bd7c097eef44b2a35d3adc581e0cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:16.134306 16251 raft_consensus.cc:2258] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.134380 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet b178ad9ad6e54a29ad0624ef74644275
I20240501 08:12:16.134485 16473 leader_election.cc:288] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:16.135195 16251 ts_tablet_manager.cc:1875] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.135010 16498 tablet_replica.cc:330] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:16.135480 16498 raft_consensus.cc:2229] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.135761 16498 raft_consensus.cc:2258] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.136375 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet b178ad9ad6e54a29ad0624ef74644275 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:16.137218 16498 ts_tablet_manager.cc:1875] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.137603 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet b178ad9ad6e54a29ad0624ef74644275 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:16.137971 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet b3fcc47f0a244a08b3adb00faa790da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:16.138099 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 76bd7c097eef44b2a35d3adc581e0cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:16.138375 16248 tablet_replica.cc:330] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:16.138535 13083 tablet_service.cc:1514] Processing DeleteTablet for tablet b3fcc47f0a244a08b3adb00faa790da6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:16.138581 16248 raft_consensus.cc:2229] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:16.138763 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet b178ad9ad6e54a29ad0624ef74644275 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:16.138938 16248 raft_consensus.cc:2258] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.138955 16250 tablet_replica.cc:330] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:16.139169 16250 raft_consensus.cc:2229] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.139390 16250 raft_consensus.cc:2258] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.139670 16248 ts_tablet_manager.cc:1875] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.140115 16250 ts_tablet_manager.cc:1875] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.140458 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 76bd7c097eef44b2a35d3adc581e0cdf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:16.144682 16498 ts_tablet_manager.cc:1888] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.144856 16498 log.cc:1196] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/76bd7c097eef44b2a35d3adc581e0cdf
I20240501 08:12:16.145352 16498 ts_tablet_manager.cc:1909] T 76bd7c097eef44b2a35d3adc581e0cdf P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:16.146365 16498 tablet_replica.cc:330] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:16.146625 16498 raft_consensus.cc:2229] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.146659 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbaf0a39a37a498b8f341cf1f643fa59" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:16.146665 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbaf0a39a37a498b8f341cf1f643fa59" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:16.146869 16498 raft_consensus.cc:2258] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.146865 13364 raft_consensus.cc:3037] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.146895 16024 raft_consensus.cc:3037] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.147483 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 76bd7c097eef44b2a35d3adc581e0cdf (table rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]) successfully deleted
I20240501 08:12:16.147616 16498 ts_tablet_manager.cc:1875] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.150132 16024 raft_consensus.cc:2454] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:16.150626 16092 leader_election.cc:302] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:16.151006 16473 raft_consensus.cc:2781] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.151177 16473 raft_consensus.cc:686] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:16.151445 16473 consensus_queue.cc:237] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.154454 16498 ts_tablet_manager.cc:1888] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.154640 16498 log.cc:1196] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/b178ad9ad6e54a29ad0624ef74644275
I20240501 08:12:16.155097 16498 ts_tablet_manager.cc:1909] T b178ad9ad6e54a29ad0624ef74644275 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:16.155378 13364 raft_consensus.cc:2454] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:16.157040 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet b178ad9ad6e54a29ad0624ef74644275 (table rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]) successfully deleted
I20240501 08:12:16.159554 16248 ts_tablet_manager.cc:1888] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.159937 16289 consensus_queue.cc:1035] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.160943 16250 ts_tablet_manager.cc:1888] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.161015 16251 ts_tablet_manager.cc:1888] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.161327 16250 log.cc:1196] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/b3fcc47f0a244a08b3adb00faa790da6
I20240501 08:12:16.161356 16251 log.cc:1196] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/b3fcc47f0a244a08b3adb00faa790da6
I20240501 08:12:16.161545 16248 log.cc:1196] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/b3fcc47f0a244a08b3adb00faa790da6
I20240501 08:12:16.161860 16250 ts_tablet_manager.cc:1909] T b3fcc47f0a244a08b3adb00faa790da6 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:16.162294 16251 ts_tablet_manager.cc:1909] T b3fcc47f0a244a08b3adb00faa790da6 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:16.162760 16248 ts_tablet_manager.cc:1909] T b3fcc47f0a244a08b3adb00faa790da6 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:16.162950 16250 tablet_replica.cc:330] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:16.163185 16250 raft_consensus.cc:2229] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:16.163216 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet b3fcc47f0a244a08b3adb00faa790da6 (table rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]) successfully deleted
I20240501 08:12:16.164963 12828 catalog_manager.cc:5484] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:16.166042 16250 raft_consensus.cc:2258] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.166853 16250 ts_tablet_manager.cc:1875] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.174197 16250 ts_tablet_manager.cc:1888] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.174486 16250 log.cc:1196] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/b178ad9ad6e54a29ad0624ef74644275
I20240501 08:12:16.175068 16250 ts_tablet_manager.cc:1909] T b178ad9ad6e54a29ad0624ef74644275 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:16.176234 16250 tablet_replica.cc:330] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:16.176188 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet b178ad9ad6e54a29ad0624ef74644275 (table rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]) successfully deleted
I20240501 08:12:16.176585 16250 raft_consensus.cc:2229] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.176892 16250 raft_consensus.cc:2258] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.177937 16250 ts_tablet_manager.cc:1875] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.178267 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet b3fcc47f0a244a08b3adb00faa790da6 (table rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]) successfully deleted
I20240501 08:12:16.179704 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet b3fcc47f0a244a08b3adb00faa790da6 (table rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]) successfully deleted
I20240501 08:12:16.180109 16473 consensus_queue.cc:1035] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.181818 16248 tablet_replica.cc:330] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:16.182405 16251 tablet_replica.cc:330] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:16.182926 16251 raft_consensus.cc:2229] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.183465 16248 raft_consensus.cc:2229] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:16.184125 16251 raft_consensus.cc:2258] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.184864 16248 raft_consensus.cc:2258] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.184856 16250 ts_tablet_manager.cc:1888] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.185052 16250 log.cc:1196] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/76bd7c097eef44b2a35d3adc581e0cdf
I20240501 08:12:16.185633 16250 ts_tablet_manager.cc:1909] T 76bd7c097eef44b2a35d3adc581e0cdf P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:16.186466 16251 ts_tablet_manager.cc:1875] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.187057 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 76bd7c097eef44b2a35d3adc581e0cdf (table rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]) successfully deleted
I20240501 08:12:16.188928 16248 ts_tablet_manager.cc:1875] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.203845 16251 ts_tablet_manager.cc:1888] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.203830 16248 ts_tablet_manager.cc:1888] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.203989 16251 log.cc:1196] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/b178ad9ad6e54a29ad0624ef74644275
I20240501 08:12:16.204008 16248 log.cc:1196] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/76bd7c097eef44b2a35d3adc581e0cdf
I20240501 08:12:16.204481 16248 ts_tablet_manager.cc:1909] T 76bd7c097eef44b2a35d3adc581e0cdf P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:16.204493 16251 ts_tablet_manager.cc:1909] T b178ad9ad6e54a29ad0624ef74644275 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:16.205549 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet b178ad9ad6e54a29ad0624ef74644275 (table rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]) successfully deleted
I20240501 08:12:16.205967 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 76bd7c097eef44b2a35d3adc581e0cdf (table rebalance_test_table_8 [id=0f0bc268a64d4bee90527ed42efaf03d]) successfully deleted
I20240501 08:12:16.236258 13364 consensus_queue.cc:237] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.239058 13235 raft_consensus.cc:1261] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:16.239542 13104 raft_consensus.cc:1261] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:16.240132 16497 consensus_queue.cc:1035] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:16.240512 15502 consensus_queue.cc:1035] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:16.250083 13298 consensus_peers.cc:485] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a54efcd41aeb4c24adfa51bb543d9661. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:16.251772 15502 raft_consensus.cc:2932] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:16.252523 13104 raft_consensus.cc:2932] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:16.257922 13235 raft_consensus.cc:2932] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:16.264247 12832 catalog_manager.cc:5484] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:16.266218 16481 consensus_queue.cc:1035] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.275970 16481 consensus_queue.cc:1035] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.300967 13105 consensus_queue.cc:237] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.304248 13235 raft_consensus.cc:1261] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:16.304791 13364 raft_consensus.cc:1261] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:16.305375 16546 consensus_queue.cc:1035] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:16.305953 15652 consensus_queue.cc:1035] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:16.313077 16334 consensus_queue.cc:1035] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:16.313220 13040 consensus_peers.cc:485] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 911cc47749484e88882c0efa75fc6396. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:16.319344 15652 raft_consensus.cc:2932] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:16.320237 13364 raft_consensus.cc:2932] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:16.326208 16334 consensus_queue.cc:1035] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.327845 13235 raft_consensus.cc:2932] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:16.328553 12824 catalog_manager.cc:5484] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:16.363931 12828 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:16.370766 13364 raft_consensus.cc:1261] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:16.370818 16024 raft_consensus.cc:1261] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:16.371430 16473 consensus_queue.cc:1035] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.371707 16289 consensus_queue.cc:1035] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:16.384547 16502 mvcc.cc:205] Tried to move back new op lower bound from 7022801454568611840 to 7022801453679218688. Current Snapshot: MvccSnapshot[applied={T|T < 7022801454568611840}]
I20240501 08:12:16.384893 16572 tablet.cc:1722] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:16.385432 16572 tablet.cc:1620] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:16.383370 12828 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet fbaf0a39a37a498b8f341cf1f643fa59 (table rebalancer_dynamic_table [id=16dfc2830de84c11b1903858a11263a3]). Expected version 1 got 0
I20240501 08:12:16.389803 16573 tablet.cc:1722] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:16.390100 12828 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet fbaf0a39a37a498b8f341cf1f643fa59 (table rebalancer_dynamic_table [id=16dfc2830de84c11b1903858a11263a3]). Expected version 1 got 0
I20240501 08:12:16.390516 16573 tablet.cc:1620] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:16.394132 12972 consensus_queue.cc:237] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:16.396971 12828 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=16dfc2830de84c11b1903858a11263a3] alter complete (version 1)
I20240501 08:12:16.399500 16574 tablet.cc:1722] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:16.400812 16574 tablet.cc:1620] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:16.402698 12824 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet fbaf0a39a37a498b8f341cf1f643fa59 (table rebalancer_dynamic_table [id=16dfc2830de84c11b1903858a11263a3]). Expected version 1 got 0
I20240501 08:12:16.402766 13235 raft_consensus.cc:1261] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:16.403357 16075 consensus_queue.cc:1035] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:16.407507 13105 raft_consensus.cc:1261] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:16.411116 12909 consensus_peers.cc:485] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba940b0d8b064284bfc71952801d17c3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:16.411701 16578 consensus_queue.cc:1035] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:16.430238 16578 raft_consensus.cc:2932] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:16.430989 13105 raft_consensus.cc:2932] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:16.437747 13235 raft_consensus.cc:2932] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:16.439067 12824 catalog_manager.cc:5484] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:16.490556 13364 consensus_queue.cc:237] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.493103 12972 raft_consensus.cc:1261] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:16.493288 13235 raft_consensus.cc:1261] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:16.494016 16379 consensus_queue.cc:1035] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:16.494591 15502 consensus_queue.cc:1035] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:16.497393 16379 raft_consensus.cc:2932] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:16.498392 13235 raft_consensus.cc:2932] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
W20240501 08:12:16.501271 13298 consensus_peers.cc:485] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3b5240d5db124f409b28fbde8cf68f8a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:16.502166 12972 raft_consensus.cc:2932] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:16.503702 12824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:16.508142 12828 catalog_manager.cc:5484] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:16.512952 12826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:16.524677 16572 tablet.cc:1722] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:16.525183 16572 tablet.cc:1620] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:16.527307 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 8a48ba4666c14848b2b5a1e62f16357b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.527885 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a48ba4666c14848b2b5a1e62f16357b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.529448 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 8a48ba4666c14848b2b5a1e62f16357b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.529640 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5c7f2d0156405da42d25d1e18ccbfa (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.529829 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 7da2efbf509e4d8c82acc7b12fc6f803 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.529938 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a48ba4666c14848b2b5a1e62f16357b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.530023 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5c7f2d0156405da42d25d1e18ccbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.530246 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da2efbf509e4d8c82acc7b12fc6f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.531934 15871 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5c7f2d0156405da42d25d1e18ccbfa (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.532457 15871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5c7f2d0156405da42d25d1e18ccbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.536768 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 7da2efbf509e4d8c82acc7b12fc6f803 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.537264 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da2efbf509e4d8c82acc7b12fc6f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.540812 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 8a48ba4666c14848b2b5a1e62f16357b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.541317 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a48ba4666c14848b2b5a1e62f16357b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.544644 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 7da2efbf509e4d8c82acc7b12fc6f803 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.545164 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da2efbf509e4d8c82acc7b12fc6f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.546733 16601 tablet_bootstrap.cc:492] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:16.547092 16573 tablet.cc:1722] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:16.547991 16573 tablet.cc:1620] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:16.554692 16603 tablet_bootstrap.cc:492] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:16.557252 16603 tablet_bootstrap.cc:654] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.558970 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=16dfc2830de84c11b1903858a11263a3] alter complete (version 2)
I20240501 08:12:16.561342 16601 tablet_bootstrap.cc:654] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.568262 16603 tablet_bootstrap.cc:492] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:16.568753 16603 ts_tablet_manager.cc:1367] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20240501 08:12:16.570075 16603 raft_consensus.cc:348] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.571501 16601 tablet_bootstrap.cc:492] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:16.571717 16601 ts_tablet_manager.cc:1367] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.004s
I20240501 08:12:16.572602 16601 raft_consensus.cc:348] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.572875 16601 raft_consensus.cc:374] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.573014 16601 raft_consensus.cc:729] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.573366 16601 consensus_queue.cc:260] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.574653 16601 ts_tablet_manager.cc:1398] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:16.577456 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5c7f2d0156405da42d25d1e18ccbfa (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:16.579331 16603 raft_consensus.cc:374] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.580097 16603 raft_consensus.cc:729] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.580538 16603 consensus_queue.cc:260] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.581007 16574 tablet.cc:1722] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:16.581524 16574 tablet.cc:1620] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:16.582358 16603 ts_tablet_manager.cc:1398] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240501 08:12:16.583717 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5c7f2d0156405da42d25d1e18ccbfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.596971 16612 tablet_bootstrap.cc:492] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:16.599131 16612 tablet_bootstrap.cc:654] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.611404 16612 tablet_bootstrap.cc:492] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:16.611749 16612 ts_tablet_manager.cc:1367] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20240501 08:12:16.612990 16612 raft_consensus.cc:348] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.613380 16612 raft_consensus.cc:374] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.613531 16612 raft_consensus.cc:729] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.613893 16612 consensus_queue.cc:260] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.614688 16612 ts_tablet_manager.cc:1398] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:16.615180 16612 tablet_bootstrap.cc:492] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:16.615563 16334 raft_consensus.cc:482] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.615770 16334 raft_consensus.cc:504] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.616680 16334 leader_election.cc:288] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:16.617084 16602 tablet_bootstrap.cc:492] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:16.619066 16602 tablet_bootstrap.cc:654] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.619074 16612 tablet_bootstrap.cc:654] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.619794 16609 tablet_bootstrap.cc:492] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:16.620793 16610 tablet_bootstrap.cc:492] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:16.623044 16610 tablet_bootstrap.cc:654] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.632215 16609 tablet_bootstrap.cc:654] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.638975 12826 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 08:12:16.639309 12826 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240501 08:12:16.641336 16610 tablet_bootstrap.cc:492] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:16.641541 16610 ts_tablet_manager.cc:1367] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 08:12:16.642658 16610 raft_consensus.cc:348] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.642993 16610 raft_consensus.cc:374] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.643136 16610 raft_consensus.cc:729] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.643512 16610 consensus_queue.cc:260] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.644785 16609 tablet_bootstrap.cc:492] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:16.645013 16609 ts_tablet_manager.cc:1367] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240501 08:12:16.646229 16609 raft_consensus.cc:348] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.646550 16609 raft_consensus.cc:374] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.646749 16609 raft_consensus.cc:729] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.650187 12826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 73fafaa4edb748c9ab9d36b519ff9b07
I20240501 08:12:16.651422 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 73fafaa4edb748c9ab9d36b519ff9b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:16.652200 12826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 4a39a9ed18924e7b88d4ae5a516105a8
I20240501 08:12:16.652503 16250 tablet_replica.cc:330] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:16.652822 16250 raft_consensus.cc:2229] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:16.652984 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet 73fafaa4edb748c9ab9d36b519ff9b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:16.653067 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a39a9ed18924e7b88d4ae5a516105a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:16.653303 16250 raft_consensus.cc:2258] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.653286 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 73fafaa4edb748c9ab9d36b519ff9b07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:16.653808 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a39a9ed18924e7b88d4ae5a516105a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:16.653864 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a39a9ed18924e7b88d4ae5a516105a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:16.654063 12826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 3720cbc21425459d9f0cb91cf825e521
I20240501 08:12:16.654100 16251 tablet_replica.cc:330] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:16.654222 16250 ts_tablet_manager.cc:1875] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.654337 16251 raft_consensus.cc:2229] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.654569 16251 raft_consensus.cc:2258] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.654808 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 3720cbc21425459d9f0cb91cf825e521 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:16.654850 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 3720cbc21425459d9f0cb91cf825e521 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:16.655290 16251 ts_tablet_manager.cc:1875] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.655519 16498 tablet_replica.cc:330] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:16.655580 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet 3720cbc21425459d9f0cb91cf825e521 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:16.655755 16498 raft_consensus.cc:2229] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.656016 16498 raft_consensus.cc:2258] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.656729 16498 ts_tablet_manager.cc:1875] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.659998 16248 tablet_replica.cc:330] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:16.660228 16248 raft_consensus.cc:2229] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.660487 16248 raft_consensus.cc:2258] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.661667 16251 ts_tablet_manager.cc:1888] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.661701 16248 ts_tablet_manager.cc:1875] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.661979 16251 log.cc:1196] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/4a39a9ed18924e7b88d4ae5a516105a8
I20240501 08:12:16.662503 16251 ts_tablet_manager.cc:1909] T 4a39a9ed18924e7b88d4ae5a516105a8 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:16.663794 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 4a39a9ed18924e7b88d4ae5a516105a8 (table rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]) successfully deleted
I20240501 08:12:16.665570 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:16.668164 16498 ts_tablet_manager.cc:1888] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.680099 16572 tablet.cc:1722] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:16.682566 16248 ts_tablet_manager.cc:1888] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.682816 16248 log.cc:1196] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/73fafaa4edb748c9ab9d36b519ff9b07
I20240501 08:12:16.683387 16248 ts_tablet_manager.cc:1909] T 73fafaa4edb748c9ab9d36b519ff9b07 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:16.684664 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 73fafaa4edb748c9ab9d36b519ff9b07 (table rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]) successfully deleted
I20240501 08:12:16.685333 16250 ts_tablet_manager.cc:1888] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.685356 16573 tablet.cc:1722] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:16.685549 16250 log.cc:1196] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/73fafaa4edb748c9ab9d36b519ff9b07
I20240501 08:12:16.686061 16250 ts_tablet_manager.cc:1909] T 73fafaa4edb748c9ab9d36b519ff9b07 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:16.687263 16250 tablet_replica.cc:330] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:16.687481 16250 raft_consensus.cc:2229] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.687731 16250 raft_consensus.cc:2258] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.688462 16250 ts_tablet_manager.cc:1875] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.688463 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 73fafaa4edb748c9ab9d36b519ff9b07 (table rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]) successfully deleted
I20240501 08:12:16.691035 16498 log.cc:1196] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/73fafaa4edb748c9ab9d36b519ff9b07
I20240501 08:12:16.692013 16498 ts_tablet_manager.cc:1909] T 73fafaa4edb748c9ab9d36b519ff9b07 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:16.692433 16574 tablet.cc:1722] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:16.693223 16498 tablet_replica.cc:330] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:16.693492 16498 raft_consensus.cc:2229] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:16.693485 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 73fafaa4edb748c9ab9d36b519ff9b07 (table rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]) successfully deleted
I20240501 08:12:16.693928 16498 raft_consensus.cc:2258] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.694831 16498 ts_tablet_manager.cc:1875] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.701051 16602 tablet_bootstrap.cc:492] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:16.701342 16602 ts_tablet_manager.cc:1367] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.084s user 0.004s sys 0.000s
I20240501 08:12:16.702294 16602 raft_consensus.cc:348] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.702601 16602 raft_consensus.cc:374] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.702754 16602 raft_consensus.cc:729] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.703390 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=16dfc2830de84c11b1903858a11263a3] alter complete (version 3)
I20240501 08:12:16.703109 16602 consensus_queue.cc:260] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.704543 16602 ts_tablet_manager.cc:1398] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:16.705824 16602 tablet_bootstrap.cc:492] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:16.707667 16602 tablet_bootstrap.cc:654] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.708916 16248 tablet_replica.cc:330] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:16.709334 16248 raft_consensus.cc:2229] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:16.709874 16248 raft_consensus.cc:2258] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.712756 16248 ts_tablet_manager.cc:1875] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.716585 16609 consensus_queue.cc:260] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.718065 16609 ts_tablet_manager.cc:1398] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.073s user 0.002s sys 0.000s
I20240501 08:12:16.718550 16609 tablet_bootstrap.cc:492] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:16.720439 16609 tablet_bootstrap.cc:654] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:16.648342 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a48ba4666c14848b2b5a1e62f16357b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:16.720898 15756 raft_consensus.cc:2454] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:16.721447 15959 leader_election.cc:302] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:16.721959 16610 ts_tablet_manager.cc:1398] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.080s user 0.003s sys 0.000s
I20240501 08:12:16.722680 16334 raft_consensus.cc:2781] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.722785 16334 raft_consensus.cc:482] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.722874 16334 raft_consensus.cc:3037] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.725865 16334 raft_consensus.cc:504] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.727519 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a48ba4666c14848b2b5a1e62f16357b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:16.727528 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a48ba4666c14848b2b5a1e62f16357b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:16.728709 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a48ba4666c14848b2b5a1e62f16357b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:16.728917 15756 raft_consensus.cc:3037] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.731748 15756 raft_consensus.cc:2454] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:16.732453 15959 leader_election.cc:302] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
W20240501 08:12:16.732858 15959 leader_election.cc:341] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:12:16.733346 15959 leader_election.cc:341] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:16.733515 16612 tablet_bootstrap.cc:492] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:16.733563 16367 raft_consensus.cc:2781] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.733696 16612 ts_tablet_manager.cc:1367] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.119s user 0.050s sys 0.045s
I20240501 08:12:16.734530 16612 raft_consensus.cc:348] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.734755 16612 raft_consensus.cc:374] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.734823 16612 raft_consensus.cc:729] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.735105 16612 consensus_queue.cc:260] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.735663 16612 ts_tablet_manager.cc:1398] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:16.741662 15502 raft_consensus.cc:482] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.741855 15502 raft_consensus.cc:504] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.742486 15502 leader_election.cc:288] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:16.743096 16602 tablet_bootstrap.cc:492] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:16.743376 16602 ts_tablet_manager.cc:1367] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240501 08:12:16.743356 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da2efbf509e4d8c82acc7b12fc6f803" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:16.743386 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da2efbf509e4d8c82acc7b12fc6f803" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:16.743585 15893 raft_consensus.cc:2454] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:16.743625 16158 raft_consensus.cc:2454] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:16.744208 13298 leader_election.cc:302] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:16.744460 16602 raft_consensus.cc:348] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.745077 16602 raft_consensus.cc:374] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.745246 16602 raft_consensus.cc:729] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.745766 15502 raft_consensus.cc:2781] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.745646 16602 consensus_queue.cc:260] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.745858 15502 raft_consensus.cc:482] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.745954 15502 raft_consensus.cc:3037] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.746482 16602 ts_tablet_manager.cc:1398] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:16.751295 16248 ts_tablet_manager.cc:1888] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.751499 16248 log.cc:1196] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/3720cbc21425459d9f0cb91cf825e521
I20240501 08:12:16.752063 16248 ts_tablet_manager.cc:1909] T 3720cbc21425459d9f0cb91cf825e521 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:16.754480 16367 raft_consensus.cc:686] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:16.754828 16367 consensus_queue.cc:237] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:16.754984 16250 ts_tablet_manager.cc:1888] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.755182 16250 log.cc:1196] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/4a39a9ed18924e7b88d4ae5a516105a8
I20240501 08:12:16.761242 15502 raft_consensus.cc:504] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.762202 16334 leader_election.cc:288] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:16.762441 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da2efbf509e4d8c82acc7b12fc6f803" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:16.762701 16158 raft_consensus.cc:3037] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.763253 15502 leader_election.cc:288] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:16.765223 16498 ts_tablet_manager.cc:1888] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.765781 16158 raft_consensus.cc:2454] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:16.766597 16609 tablet_bootstrap.cc:492] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:16.766700 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da2efbf509e4d8c82acc7b12fc6f803" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:16.766839 16609 ts_tablet_manager.cc:1367] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20240501 08:12:16.766940 15893 raft_consensus.cc:3037] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.768064 12828 catalog_manager.cc:5484] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:16.768200 16609 raft_consensus.cc:348] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.768589 16609 raft_consensus.cc:374] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:16.768692 16609 raft_consensus.cc:729] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:16.769060 16609 consensus_queue.cc:260] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.769980 16609 ts_tablet_manager.cc:1398] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:16.770378 15893 raft_consensus.cc:2454] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:16.770586 13298 leader_election.cc:302] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:16.770841 16250 ts_tablet_manager.cc:1909] T 4a39a9ed18924e7b88d4ae5a516105a8 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:16.771885 16379 raft_consensus.cc:2781] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.772092 16379 raft_consensus.cc:686] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:16.772475 16379 consensus_queue.cc:237] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:16.773548 16627 ts_tablet_manager.cc:902] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:16.773783 16498 log.cc:1196] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/4a39a9ed18924e7b88d4ae5a516105a8
I20240501 08:12:16.775353 16498 ts_tablet_manager.cc:1909] T 4a39a9ed18924e7b88d4ae5a516105a8 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:16.775352 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 4a39a9ed18924e7b88d4ae5a516105a8 (table rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]) successfully deleted
I20240501 08:12:16.777861 16250 tablet_replica.cc:330] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:16.778100 16250 raft_consensus.cc:2229] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.778409 16250 raft_consensus.cc:2258] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.779171 16250 ts_tablet_manager.cc:1875] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.779210 12828 catalog_manager.cc:5484] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:16.779632 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 4a39a9ed18924e7b88d4ae5a516105a8 (table rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]) successfully deleted
I20240501 08:12:16.780540 16498 tablet_replica.cc:330] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:16.781203 16498 raft_consensus.cc:2229] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.782236 16498 raft_consensus.cc:2258] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.783802 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 3720cbc21425459d9f0cb91cf825e521 (table rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]) successfully deleted
I20240501 08:12:16.784632 16498 ts_tablet_manager.cc:1875] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.785914 16250 ts_tablet_manager.cc:1888] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.786090 16250 log.cc:1196] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/3720cbc21425459d9f0cb91cf825e521
I20240501 08:12:16.787513 16250 ts_tablet_manager.cc:1909] T 3720cbc21425459d9f0cb91cf825e521 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:16.788763 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 3720cbc21425459d9f0cb91cf825e521 (table rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]) successfully deleted
I20240501 08:12:16.791216 16627 tablet_copy_client.cc:308] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:12:16.792516 13125 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet 911cc47749484e88882c0efa75fc6396 from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:42231)
I20240501 08:12:16.792778 13125 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet 911cc47749484e88882c0efa75fc6396 from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:42231: session id = 5dd722c770d34d6c9154a87917117161-911cc47749484e88882c0efa75fc6396
I20240501 08:12:16.793311 16498 ts_tablet_manager.cc:1888] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:16.793458 16498 log.cc:1196] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/3720cbc21425459d9f0cb91cf825e521
I20240501 08:12:16.793963 16498 ts_tablet_manager.cc:1909] T 3720cbc21425459d9f0cb91cf825e521 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:16.795087 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 3720cbc21425459d9f0cb91cf825e521 (table rebalance_test_table_10 [id=59c66af77cf24d538a3b224c075adb1f]) successfully deleted
I20240501 08:12:16.797266 13125 tablet_copy_source_session.cc:205] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:16.799809 16627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911cc47749484e88882c0efa75fc6396. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.809546 16627 tablet_copy_client.cc:783] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:16.810065 16627 tablet_copy_client.cc:647] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:16.813004 16627 tablet_copy_client.cc:523] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:16.815282 16481 raft_consensus.cc:482] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:16.815440 16481 raft_consensus.cc:504] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.816205 16481 leader_election.cc:288] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:16.817039 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5c7f2d0156405da42d25d1e18ccbfa" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:16.817299 16158 raft_consensus.cc:2454] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:16.817456 16627 tablet_bootstrap.cc:492] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:16.817747 15823 leader_election.cc:302] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:16.818621 16481 raft_consensus.cc:2781] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:16.818766 16481 raft_consensus.cc:482] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:16.818877 16481 raft_consensus.cc:3037] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.822708 16481 raft_consensus.cc:504] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.824704 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5c7f2d0156405da42d25d1e18ccbfa" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:16.824987 16158 raft_consensus.cc:3037] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.832687 16158 raft_consensus.cc:2454] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:16.839576 15823 leader_election.cc:302] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:16.842653 16481 leader_election.cc:288] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:16.847733 16481 raft_consensus.cc:2781] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:16.848443 16481 raft_consensus.cc:686] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:16.849073 16481 consensus_queue.cc:237] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:16.869199 12828 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:16.869457 12828 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:16.873103 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet fbaf0a39a37a498b8f341cf1f643fa59
I20240501 08:12:16.874495 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet fbaf0a39a37a498b8f341cf1f643fa59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:16.874801 16251 tablet_replica.cc:330] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:16.875020 16251 raft_consensus.cc:2229] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.875241 16251 raft_consensus.cc:2258] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.876003 16137 tablet_service.cc:1514] Processing DeleteTablet for tablet fbaf0a39a37a498b8f341cf1f643fa59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:16.876616 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet fbaf0a39a37a498b8f341cf1f643fa59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:16 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:16.877416 16251 ts_tablet_manager.cc:1875] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.879976 16646 ts_tablet_manager.cc:902] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:16.891510 16251 ts_tablet_manager.cc:1888] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:16.898530 16251 log.cc:1196] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/fbaf0a39a37a498b8f341cf1f643fa59
I20240501 08:12:16.899542 16251 ts_tablet_manager.cc:1909] T fbaf0a39a37a498b8f341cf1f643fa59 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:16.903375 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet fbaf0a39a37a498b8f341cf1f643fa59 (table rebalancer_dynamic_table_new_name [id=16dfc2830de84c11b1903858a11263a3]) successfully deleted
I20240501 08:12:16.934330 16648 tablet_replica.cc:330] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:16.934634 16648 raft_consensus.cc:2229] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:16.934891 16648 raft_consensus.cc:2258] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 08:12:16.935407 13040 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 49948.3us
I20240501 08:12:16.940235 16647 tablet_replica.cc:330] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:16.940493 16647 raft_consensus.cc:2229] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:16.941229 16647 raft_consensus.cc:2258] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:16.945187 16648 ts_tablet_manager.cc:1875] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240501 08:12:16.948438 15823 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 108934us
I20240501 08:12:16.949872 12828 catalog_manager.cc:5484] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:16.952502 16647 ts_tablet_manager.cc:1875] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:16.956662 16643 ts_tablet_manager.cc:902] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:16.970221 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5c7f2d0156405da42d25d1e18ccbfa" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:16.970556 13105 raft_consensus.cc:2454] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:16.971575 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5c7f2d0156405da42d25d1e18ccbfa" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:16.971763 13105 raft_consensus.cc:3037] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:16.974697 13105 raft_consensus.cc:2454] T 6c5c7f2d0156405da42d25d1e18ccbfa P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:16.976187 12828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:16.986502 16646 tablet_copy_client.cc:308] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:12:16.987320 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd45ed878394e30b1f6d5fb7ccff036 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8a9ac4cc66e40afa62d7f2156c258f4]), partition=
I20240501 08:12:16.987373 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd45ed878394e30b1f6d5fb7ccff036 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8a9ac4cc66e40afa62d7f2156c258f4]), partition=
I20240501 08:12:16.987767 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd45ed878394e30b1f6d5fb7ccff036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.987758 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd45ed878394e30b1f6d5fb7ccff036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.993175 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 7dd45ed878394e30b1f6d5fb7ccff036 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b8a9ac4cc66e40afa62d7f2156c258f4]), partition=
I20240501 08:12:16.993578 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dd45ed878394e30b1f6d5fb7ccff036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:16.993599 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet ba940b0d8b064284bfc71952801d17c3 from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:47087)
I20240501 08:12:16.993811 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet ba940b0d8b064284bfc71952801d17c3 from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:47087: session id = 982f49fc87f149439cda0de2261b6bfd-ba940b0d8b064284bfc71952801d17c3
I20240501 08:12:16.994388 16648 ts_tablet_manager.cc:1888] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:16.994526 16648 log.cc:1196] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/fbaf0a39a37a498b8f341cf1f643fa59
I20240501 08:12:16.994953 16648 ts_tablet_manager.cc:1909] T fbaf0a39a37a498b8f341cf1f643fa59 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:16.997185 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet fbaf0a39a37a498b8f341cf1f643fa59 (table rebalancer_dynamic_table_new_name [id=16dfc2830de84c11b1903858a11263a3]) successfully deleted
I20240501 08:12:16.997447 12994 tablet_copy_source_session.cc:205] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:16.999339 16647 ts_tablet_manager.cc:1888] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:16.999766 16647 log.cc:1196] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/fbaf0a39a37a498b8f341cf1f643fa59
I20240501 08:12:17.000473 16646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba940b0d8b064284bfc71952801d17c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.000480 16647 ts_tablet_manager.cc:1909] T fbaf0a39a37a498b8f341cf1f643fa59 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:17.002739 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet fbaf0a39a37a498b8f341cf1f643fa59 (table rebalancer_dynamic_table_new_name [id=16dfc2830de84c11b1903858a11263a3]) successfully deleted
I20240501 08:12:17.009943 16601 tablet_bootstrap.cc:492] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:17.011472 16601 tablet_bootstrap.cc:654] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.015319 16609 tablet_bootstrap.cc:492] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:17.016269 16610 tablet_bootstrap.cc:492] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:17.017071 16609 tablet_bootstrap.cc:654] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.017773 16610 tablet_bootstrap.cc:654] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.022344 16646 tablet_copy_client.cc:783] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:17.023155 16646 tablet_copy_client.cc:647] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:17.033538 16610 tablet_bootstrap.cc:492] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:17.033903 16610 ts_tablet_manager.cc:1367] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.008s
I20240501 08:12:17.034919 16610 raft_consensus.cc:348] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.035301 16610 raft_consensus.cc:374] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.035475 16610 raft_consensus.cc:729] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.035902 16610 consensus_queue.cc:260] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.036603 16609 tablet_bootstrap.cc:492] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:17.036891 16609 ts_tablet_manager.cc:1367] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20240501 08:12:17.037606 16646 tablet_copy_client.cc:523] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:17.038020 16609 raft_consensus.cc:348] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.038331 16609 raft_consensus.cc:374] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.038388 16610 ts_tablet_manager.cc:1398] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:17.038486 16609 raft_consensus.cc:729] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.039227 16609 consensus_queue.cc:260] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.040745 16609 ts_tablet_manager.cc:1398] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:17.046542 16646 tablet_bootstrap.cc:492] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:17.056546 16661 ts_tablet_manager.cc:902] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:17.057265 16601 tablet_bootstrap.cc:492] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:17.057571 16601 ts_tablet_manager.cc:1367] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.000s
I20240501 08:12:17.058604 16601 raft_consensus.cc:348] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.058934 16601 raft_consensus.cc:374] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.059386 16601 raft_consensus.cc:729] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.059732 16601 consensus_queue.cc:260] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.061923 16601 ts_tablet_manager.cc:1398] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:17.078047 16643 tablet_copy_client.cc:308] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:12:17.095784 16661 tablet_copy_client.cc:308] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:12:17.097470 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 3b5240d5db124f409b28fbde8cf68f8a from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:50899)
I20240501 08:12:17.097685 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 3b5240d5db124f409b28fbde8cf68f8a from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:50899: session id = 6cc32f6e2d49477f9a23ce6e444039fb-3b5240d5db124f409b28fbde8cf68f8a
I20240501 08:12:17.101540 13385 tablet_copy_source_session.cc:205] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:17.104566 16661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5240d5db124f409b28fbde8cf68f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.113940 16661 tablet_copy_client.cc:783] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:17.114485 16661 tablet_copy_client.cc:647] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:17.118770 16661 tablet_copy_client.cc:523] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:17.123050 16661 tablet_bootstrap.cc:492] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:17.130566 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet a54efcd41aeb4c24adfa51bb543d9661 from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:45907)
I20240501 08:12:17.130790 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet a54efcd41aeb4c24adfa51bb543d9661 from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:45907: session id = 4d516916eeac468db514eeb928f06190-a54efcd41aeb4c24adfa51bb543d9661
I20240501 08:12:17.134130 13385 tablet_copy_source_session.cc:205] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:17.137856 16643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54efcd41aeb4c24adfa51bb543d9661. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.150202 16367 raft_consensus.cc:482] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.150393 16367 raft_consensus.cc:504] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.151139 16367 leader_election.cc:288] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:17.152194 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd45ed878394e30b1f6d5fb7ccff036" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:17.152487 15756 raft_consensus.cc:2454] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:17.152902 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd45ed878394e30b1f6d5fb7ccff036" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:17.153154 15893 raft_consensus.cc:2454] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:17.153947 15957 leader_election.cc:302] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:17.154464 16334 raft_consensus.cc:2781] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:17.154577 16334 raft_consensus.cc:482] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:17.154664 16334 raft_consensus.cc:3037] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.156893 12832 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 08:12:17.157132 13105 raft_consensus.cc:3037] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.157163 12832 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240501 08:12:17.163865 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet eaf75b8d04534d898e233761a7f6f299
I20240501 08:12:17.164232 16627 tablet_bootstrap.cc:492] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.164311 16624 consensus_queue.cc:1035] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 08:12:17.164795 16627 tablet_bootstrap.cc:492] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:12:17.164979 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet eaf75b8d04534d898e233761a7f6f299 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:17.165289 16627 ts_tablet_manager.cc:1367] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.348s user 0.065s sys 0.027s
I20240501 08:12:17.165340 16250 tablet_replica.cc:330] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:17.165596 16250 raft_consensus.cc:2229] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.165867 16250 raft_consensus.cc:2258] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.166108 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet eaf75b8d04534d898e233761a7f6f299 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:17.166143 16627 raft_consensus.cc:348] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:17.166401 16627 raft_consensus.cc:729] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.166427 16251 tablet_replica.cc:330] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:17.166682 16250 ts_tablet_manager.cc:1875] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.166729 16251 raft_consensus.cc:2229] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.166640 16627 consensus_queue.cc:260] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:17.167029 16251 raft_consensus.cc:2258] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.167218 16627 ts_tablet_manager.cc:1398] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:17.168012 16251 ts_tablet_manager.cc:1875] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.169303 16646 tablet_bootstrap.cc:492] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.169770 16646 tablet_bootstrap.cc:492] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:12:17.170044 16646 ts_tablet_manager.cc:1367] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.124s user 0.049s sys 0.010s
I20240501 08:12:17.170920 16646 raft_consensus.cc:348] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:17.171173 16646 raft_consensus.cc:729] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.171478 16646 consensus_queue.cc:260] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:17.172101 16646 ts_tablet_manager.cc:1398] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:17.173099 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-ba940b0d8b064284bfc71952801d17c3 received from {username='slave'} at 127.6.56.69:47087
I20240501 08:12:17.173218 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet eaf75b8d04534d898e233761a7f6f299 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:17.173312 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-ba940b0d8b064284bfc71952801d17c3 on tablet ba940b0d8b064284bfc71952801d17c3 with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:17.173530 16248 tablet_replica.cc:330] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:17.174867 13125 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-911cc47749484e88882c0efa75fc6396 received from {username='slave'} at 127.6.56.71:42231
I20240501 08:12:17.175076 13125 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 5dd722c770d34d6c9154a87917117161-911cc47749484e88882c0efa75fc6396 on tablet 911cc47749484e88882c0efa75fc6396 with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:12:17.175128 16334 raft_consensus.cc:504] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.176276 16248 raft_consensus.cc:2229] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:17.176687 16248 raft_consensus.cc:2258] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.176743 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd45ed878394e30b1f6d5fb7ccff036" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:17.177007 15893 raft_consensus.cc:3037] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.177484 16248 ts_tablet_manager.cc:1875] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.178550 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet ba940b0d8b064284bfc71952801d17c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:17.178843 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 4 replicas of tablet ba940b0d8b064284bfc71952801d17c3
I20240501 08:12:17.179783 15893 raft_consensus.cc:2454] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:17.179895 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet ba940b0d8b064284bfc71952801d17c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:17.180235 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet ba6320d4451f44fb85f02e2c55393ef9
I20240501 08:12:17.180213 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet ba940b0d8b064284bfc71952801d17c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:17.180503 16498 tablet_replica.cc:330] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:17.180512 15957 leader_election.cc:302] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:17.180734 16498 raft_consensus.cc:2229] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.180830 16624 raft_consensus.cc:2781] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:17.180989 16624 raft_consensus.cc:686] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:17.181051 16498 raft_consensus.cc:2258] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.181232 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet ba940b0d8b064284bfc71952801d17c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:17.181315 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6320d4451f44fb85f02e2c55393ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:17.181278 16624 consensus_queue.cc:237] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.181640 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6320d4451f44fb85f02e2c55393ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:17.181710 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet ba940b0d8b064284bfc71952801d17c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:17.182422 16498 ts_tablet_manager.cc:1875] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.183367 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6320d4451f44fb85f02e2c55393ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:17.183733 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dd45ed878394e30b1f6d5fb7ccff036" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:17.183965 15756 raft_consensus.cc:3037] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.184468 16334 leader_election.cc:288] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:17.186867 15756 raft_consensus.cc:2454] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:17.187744 16251 ts_tablet_manager.cc:1888] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.187901 16251 log.cc:1196] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/eaf75b8d04534d898e233761a7f6f299
I20240501 08:12:17.190011 16251 ts_tablet_manager.cc:1909] T eaf75b8d04534d898e233761a7f6f299 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:17.191181 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet eaf75b8d04534d898e233761a7f6f299 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.191470 16643 tablet_copy_client.cc:783] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:17.191807 16643 tablet_copy_client.cc:647] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:17.192029 16250 ts_tablet_manager.cc:1888] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.192369 16334 consensus_queue.cc:1035] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.193192 16251 tablet_replica.cc:330] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:17.193187 16248 ts_tablet_manager.cc:1888] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.193315 16248 log.cc:1196] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/eaf75b8d04534d898e233761a7f6f299
I20240501 08:12:17.193439 16251 raft_consensus.cc:2229] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.193729 16248 ts_tablet_manager.cc:1909] T eaf75b8d04534d898e233761a7f6f299 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:17.193737 16251 raft_consensus.cc:2258] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.194474 16251 ts_tablet_manager.cc:1875] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.194476 16498 ts_tablet_manager.cc:1888] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.194664 16498 log.cc:1196] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba940b0d8b064284bfc71952801d17c3
I20240501 08:12:17.195168 16498 ts_tablet_manager.cc:1909] T ba940b0d8b064284bfc71952801d17c3 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:17.196923 16498 tablet_replica.cc:330] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:17.197046 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet ba940b0d8b064284bfc71952801d17c3 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.197947 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet eaf75b8d04534d898e233761a7f6f299 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.198359 16248 tablet_replica.cc:330] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:17.198630 16248 raft_consensus.cc:2229] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:17.199155 16248 raft_consensus.cc:2258] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.199440 16498 raft_consensus.cc:2229] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:17.199838 16498 raft_consensus.cc:2258] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.200735 15502 consensus_queue.cc:1035] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.200973 16248 ts_tablet_manager.cc:1875] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.202111 16498 ts_tablet_manager.cc:1875] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.207805 16677 tablet_replica.cc:330] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:17.208101 16677 raft_consensus.cc:2229] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.208446 16677 raft_consensus.cc:2258] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.212209 16251 ts_tablet_manager.cc:1888] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.212397 16251 log.cc:1196] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/ba6320d4451f44fb85f02e2c55393ef9
I20240501 08:12:17.213068 16251 ts_tablet_manager.cc:1909] T ba6320d4451f44fb85f02e2c55393ef9 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:17.217200 16250 log.cc:1196] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/eaf75b8d04534d898e233761a7f6f299
I20240501 08:12:17.217743 16250 ts_tablet_manager.cc:1909] T eaf75b8d04534d898e233761a7f6f299 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:17.222033 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet eaf75b8d04534d898e233761a7f6f299 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.222365 16677 ts_tablet_manager.cc:1875] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.224767 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet ba6320d4451f44fb85f02e2c55393ef9 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.225191 16250 tablet_replica.cc:330] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:17.226307 16250 raft_consensus.cc:2229] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.228565 16250 raft_consensus.cc:2258] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.228875 16643 tablet_copy_client.cc:523] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:17.232676 16250 ts_tablet_manager.cc:1875] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.236665 12832 catalog_manager.cc:5484] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:17.241906 16643 tablet_bootstrap.cc:492] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:17.242336 16379 consensus_queue.cc:1035] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.243516 16250 ts_tablet_manager.cc:1888] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.246273 16677 ts_tablet_manager.cc:1888] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.246592 16677 log.cc:1196] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/ba940b0d8b064284bfc71952801d17c3
I20240501 08:12:17.247257 16677 ts_tablet_manager.cc:1909] T ba940b0d8b064284bfc71952801d17c3 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:17.248258 16250 log.cc:1196] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba940b0d8b064284bfc71952801d17c3
I20240501 08:12:17.249967 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet ba940b0d8b064284bfc71952801d17c3 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.250128 16248 ts_tablet_manager.cc:1888] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.250277 16248 log.cc:1196] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/ba940b0d8b064284bfc71952801d17c3
I20240501 08:12:17.246685 16498 ts_tablet_manager.cc:1888] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.250942 16498 log.cc:1196] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba6320d4451f44fb85f02e2c55393ef9
I20240501 08:12:17.251394 16498 ts_tablet_manager.cc:1909] T ba6320d4451f44fb85f02e2c55393ef9 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:17.252518 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet ba6320d4451f44fb85f02e2c55393ef9 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.250711 16248 ts_tablet_manager.cc:1909] T ba940b0d8b064284bfc71952801d17c3 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
W20240501 08:12:17.254912 12819 catalog_manager.cc:4797] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): delete failed for tablet ba940b0d8b064284bfc71952801d17c3 because the tablet was not found. No further retry: Not found: Tablet not found: ba940b0d8b064284bfc71952801d17c3
I20240501 08:12:17.254881 16644 consensus_queue.cc:1035] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.259600 16250 ts_tablet_manager.cc:1909] T ba940b0d8b064284bfc71952801d17c3 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:17.261195 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet ba940b0d8b064284bfc71952801d17c3 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.263278 16250 tablet_replica.cc:330] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:17.263300 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet ba940b0d8b064284bfc71952801d17c3 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.263692 16250 raft_consensus.cc:2229] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.267117 16250 raft_consensus.cc:2258] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.268157 16250 ts_tablet_manager.cc:1875] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.275902 16644 consensus_queue.cc:1035] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.304363 16024 raft_consensus.cc:1203] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:17.311527 16661 tablet_bootstrap.cc:492] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.312448 16250 ts_tablet_manager.cc:1888] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.312570 16661 tablet_bootstrap.cc:492] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:12:17.313347 16661 ts_tablet_manager.cc:1367] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.190s user 0.064s sys 0.009s
I20240501 08:12:17.314627 16661 raft_consensus.cc:348] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.315999 16661 raft_consensus.cc:729] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.316514 16661 consensus_queue.cc:260] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.317622 16661 ts_tablet_manager.cc:1398] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 08:12:17.319002 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-3b5240d5db124f409b28fbde8cf68f8a received from {username='slave'} at 127.6.56.72:50899
I20240501 08:12:17.319160 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-3b5240d5db124f409b28fbde8cf68f8a on tablet 3b5240d5db124f409b28fbde8cf68f8a with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:17.324673 16250 log.cc:1196] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba6320d4451f44fb85f02e2c55393ef9
I20240501 08:12:17.325384 16250 ts_tablet_manager.cc:1909] T ba6320d4451f44fb85f02e2c55393ef9 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:17.327936 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet ba6320d4451f44fb85f02e2c55393ef9 (table rebalance_test_table_12 [id=e5090602461741aebf39ef301b19e3a2]) successfully deleted
I20240501 08:12:17.366645 16643 tablet_bootstrap.cc:492] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:17.367334 16643 tablet_bootstrap.cc:492] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:12:17.367197 12832 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:17.367789 16643 ts_tablet_manager.cc:1367] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.126s user 0.064s sys 0.017s
I20240501 08:12:17.372269 16643 raft_consensus.cc:348] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.373214 16643 raft_consensus.cc:729] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.373625 16643 consensus_queue.cc:260] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:17.376727 15756 raft_consensus.cc:1261] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:17.376708 15892 raft_consensus.cc:1261] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:17.377471 16624 consensus_queue.cc:1035] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.377990 16643 ts_tablet_manager.cc:1398] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20240501 08:12:17.378190 16367 consensus_queue.cc:1035] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:17.379046 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-a54efcd41aeb4c24adfa51bb543d9661 received from {username='slave'} at 127.6.56.70:45907
I20240501 08:12:17.379190 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 4d516916eeac468db514eeb928f06190-a54efcd41aeb4c24adfa51bb543d9661 on tablet a54efcd41aeb4c24adfa51bb543d9661 with peer 4d516916eeac468db514eeb928f06190
I20240501 08:12:17.401613 16710 tablet.cc:1722] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:17.406450 16710 tablet.cc:1620] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:17.415820 12828 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 7dd45ed878394e30b1f6d5fb7ccff036 (table rebalancer_dynamic_table [id=b8a9ac4cc66e40afa62d7f2156c258f4]). Expected version 1 got 0
I20240501 08:12:17.417240 12832 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 7dd45ed878394e30b1f6d5fb7ccff036 (table rebalancer_dynamic_table [id=b8a9ac4cc66e40afa62d7f2156c258f4]). Expected version 1 got 0
I20240501 08:12:17.436127 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=b8a9ac4cc66e40afa62d7f2156c258f4] alter complete (version 1)
I20240501 08:12:17.452440 16711 tablet.cc:1722] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:17.453984 16711 tablet.cc:1620] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:17.455091 12826 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 7dd45ed878394e30b1f6d5fb7ccff036 (table rebalancer_dynamic_table [id=b8a9ac4cc66e40afa62d7f2156c258f4]). Expected version 1 got 0
I20240501 08:12:17.465845 16708 tablet.cc:1722] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:17.466825 16708 tablet.cc:1620] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:17.524324 12828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:17.536855 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 69d21bf5577f4b449a691ef6234376d2 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.537364 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d21bf5577f4b449a691ef6234376d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.537235 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 69d21bf5577f4b449a691ef6234376d2 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.537812 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d21bf5577f4b449a691ef6234376d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.542797 12828 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:17.547004 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 69d21bf5577f4b449a691ef6234376d2 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.547469 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d21bf5577f4b449a691ef6234376d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.551136 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc25959b69a41c98d196ed88117b26f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.551579 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc25959b69a41c98d196ed88117b26f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.557498 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc25959b69a41c98d196ed88117b26f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.557942 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc25959b69a41c98d196ed88117b26f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.562100 16708 tablet.cc:1722] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:17.563324 16708 tablet.cc:1620] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:17.563642 16710 tablet.cc:1722] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:17.564260 16710 tablet.cc:1620] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:17.568190 16711 tablet.cc:1722] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:17.568694 16711 tablet.cc:1620] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:17.576596 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc25959b69a41c98d196ed88117b26f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.577064 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc25959b69a41c98d196ed88117b26f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.579329 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 6856858a304c4e208188897fe2ebdb33 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.579784 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6856858a304c4e208188897fe2ebdb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.584771 12828 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=b8a9ac4cc66e40afa62d7f2156c258f4] alter complete (version 2)
I20240501 08:12:17.591116 16719 tablet_bootstrap.cc:492] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:17.592211 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 6856858a304c4e208188897fe2ebdb33 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.592653 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6856858a304c4e208188897fe2ebdb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.592983 16719 tablet_bootstrap.cc:654] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.594919 16717 tablet_bootstrap.cc:492] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:17.598397 16718 tablet_bootstrap.cc:492] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:17.603818 16717 tablet_bootstrap.cc:654] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.605108 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 6856858a304c4e208188897fe2ebdb33 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:17.605448 16720 tablet_bootstrap.cc:492] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:17.605778 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6856858a304c4e208188897fe2ebdb33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.609277 16157 raft_consensus.cc:1203] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:17.615700 16718 tablet_bootstrap.cc:654] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.621727 16720 tablet_bootstrap.cc:654] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.645714 16723 tablet_bootstrap.cc:492] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:17.646319 16724 tablet_bootstrap.cc:492] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:17.648507 16724 tablet_bootstrap.cc:654] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.648586 16717 tablet_bootstrap.cc:492] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:17.649055 16717 ts_tablet_manager.cc:1367] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20240501 08:12:17.650211 16717 raft_consensus.cc:348] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:17.651615 16717 raft_consensus.cc:374] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.651898 16717 raft_consensus.cc:729] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.652578 16717 consensus_queue.cc:260] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:17.653746 16723 tablet_bootstrap.cc:654] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.656742 16717 ts_tablet_manager.cc:1398] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:12:17.662014 16717 tablet_bootstrap.cc:492] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:17.670987 16719 tablet_bootstrap.cc:492] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:17.671257 16719 ts_tablet_manager.cc:1367] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.080s user 0.033s sys 0.041s
I20240501 08:12:17.672092 16723 tablet_bootstrap.cc:492] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:17.672183 16719 raft_consensus.cc:348] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:17.672485 16719 raft_consensus.cc:374] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.672628 16719 raft_consensus.cc:729] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.672955 16719 consensus_queue.cc:260] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:17.673802 16719 ts_tablet_manager.cc:1398] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:17.674338 16719 tablet_bootstrap.cc:492] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:17.674496 16723 ts_tablet_manager.cc:1367] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.004s
I20240501 08:12:17.675758 16723 raft_consensus.cc:348] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:17.676018 16719 tablet_bootstrap.cc:654] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.676088 16723 raft_consensus.cc:374] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.676213 16723 raft_consensus.cc:729] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.676587 16723 consensus_queue.cc:260] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:17.678443 16717 tablet_bootstrap.cc:654] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.680179 16724 tablet_bootstrap.cc:492] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:17.680465 16724 ts_tablet_manager.cc:1367] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.007s
I20240501 08:12:17.681592 16724 raft_consensus.cc:348] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:17.681959 16724 raft_consensus.cc:374] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.682121 16724 raft_consensus.cc:729] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.682441 16724 consensus_queue.cc:260] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:17.684302 16723 ts_tablet_manager.cc:1398] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 08:12:17.684820 16724 ts_tablet_manager.cc:1398] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:17.685110 16719 tablet_bootstrap.cc:492] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:17.685322 16719 ts_tablet_manager.cc:1367] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240501 08:12:17.686141 16719 raft_consensus.cc:348] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.686426 16719 raft_consensus.cc:374] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.686450 12824 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 08:12:17.686568 16719 raft_consensus.cc:729] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.686691 12824 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240501 08:12:17.686961 16719 consensus_queue.cc:260] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.687531 16719 ts_tablet_manager.cc:1398] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:17.691546 12824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 172dada906f54b6c842d2dd74c32203b
I20240501 08:12:17.692124 12824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet ec027f1fcf0b4a5a87e56221f3e1e5cd
I20240501 08:12:17.692327 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 172dada906f54b6c842d2dd74c32203b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:17.692580 12824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 9e39ce53fb40456285a767ac836fae9d
I20240501 08:12:17.692718 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 172dada906f54b6c842d2dd74c32203b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:17.693279 16250 tablet_replica.cc:330] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:17.693356 16248 tablet_replica.cc:330] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:17.693490 16250 raft_consensus.cc:2229] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.693536 16248 raft_consensus.cc:2229] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.693754 16248 raft_consensus.cc:2258] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.693754 16250 raft_consensus.cc:2258] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.694474 16250 ts_tablet_manager.cc:1875] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.694474 16248 ts_tablet_manager.cc:1875] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.694743 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e39ce53fb40456285a767ac836fae9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:17.694981 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet ec027f1fcf0b4a5a87e56221f3e1e5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:17.696445 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet ec027f1fcf0b4a5a87e56221f3e1e5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:17.696465 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet 172dada906f54b6c842d2dd74c32203b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:17.696714 16251 tablet_replica.cc:330] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:17.696694 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet ec027f1fcf0b4a5a87e56221f3e1e5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:17.696918 16251 raft_consensus.cc:2229] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:17.697270 16251 raft_consensus.cc:2258] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.697624 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e39ce53fb40456285a767ac836fae9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:17.698043 16251 ts_tablet_manager.cc:1875] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.698056 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e39ce53fb40456285a767ac836fae9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:17.700461 16498 tablet_replica.cc:330] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:17.700685 16498 raft_consensus.cc:2229] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:17.701081 16498 raft_consensus.cc:2258] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.703399 16250 ts_tablet_manager.cc:1888] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.703639 16250 log.cc:1196] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/172dada906f54b6c842d2dd74c32203b
I20240501 08:12:17.704087 16473 raft_consensus.cc:482] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.704180 16498 ts_tablet_manager.cc:1875] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.704277 16250 ts_tablet_manager.cc:1909] T 172dada906f54b6c842d2dd74c32203b P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:17.704325 16473 raft_consensus.cc:504] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:17.705082 16473 leader_election.cc:288] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:17.705374 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 172dada906f54b6c842d2dd74c32203b (table rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]) successfully deleted
I20240501 08:12:17.707039 16250 tablet_replica.cc:330] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:17.708083 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc25959b69a41c98d196ed88117b26f" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:17.709048 16250 raft_consensus.cc:2229] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 08:12:17.709007 16092 leader_election.cc:341] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:17.709465 16250 raft_consensus.cc:2258] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.710817 16250 ts_tablet_manager.cc:1875] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.712591 12824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:17.717463 16251 ts_tablet_manager.cc:1888] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.724449 16498 ts_tablet_manager.cc:1888] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.725198 16248 ts_tablet_manager.cc:1888] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.725479 16251 log.cc:1196] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/ec027f1fcf0b4a5a87e56221f3e1e5cd
I20240501 08:12:17.726579 16248 log.cc:1196] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/172dada906f54b6c842d2dd74c32203b
I20240501 08:12:17.727116 16248 ts_tablet_manager.cc:1909] T 172dada906f54b6c842d2dd74c32203b P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:17.727406 16251 ts_tablet_manager.cc:1909] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:17.727643 16498 log.cc:1196] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/172dada906f54b6c842d2dd74c32203b
I20240501 08:12:17.727623 16708 tablet.cc:1722] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:17.728145 16498 ts_tablet_manager.cc:1909] T 172dada906f54b6c842d2dd74c32203b P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:17.729827 16248 tablet_replica.cc:330] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:17.730023 16717 tablet_bootstrap.cc:492] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:17.729982 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 172dada906f54b6c842d2dd74c32203b (table rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]) successfully deleted
I20240501 08:12:17.730314 16717 ts_tablet_manager.cc:1367] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.068s user 0.011s sys 0.010s
I20240501 08:12:17.730758 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet ec027f1fcf0b4a5a87e56221f3e1e5cd (table rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]) successfully deleted
I20240501 08:12:17.731627 16717 raft_consensus.cc:348] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.732017 16717 raft_consensus.cc:374] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.732138 16717 raft_consensus.cc:729] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.732569 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 172dada906f54b6c842d2dd74c32203b (table rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]) successfully deleted
I20240501 08:12:17.732522 16717 consensus_queue.cc:260] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.732731 16248 raft_consensus.cc:2229] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:17.733053 16498 tablet_replica.cc:330] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:17.733433 16248 raft_consensus.cc:2258] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.733719 16498 raft_consensus.cc:2229] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.734649 16498 raft_consensus.cc:2258] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.736531 16711 tablet.cc:1722] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:17.736821 16717 ts_tablet_manager.cc:1398] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:17.737769 16498 ts_tablet_manager.cc:1875] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.737884 16473 raft_consensus.cc:482] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.738076 16473 raft_consensus.cc:504] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.738514 16248 ts_tablet_manager.cc:1875] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.738739 16473 leader_election.cc:288] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:17.739508 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d21bf5577f4b449a691ef6234376d2" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
W20240501 08:12:17.740173 16092 leader_election.cc:341] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:17.747081 12828 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=b8a9ac4cc66e40afa62d7f2156c258f4] alter complete (version 3)
I20240501 08:12:17.748754 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc25959b69a41c98d196ed88117b26f" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:17.749037 12972 raft_consensus.cc:2454] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:17.750578 16093 leader_election.cc:302] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 76d830e199eb461c8ca689c9460ee0c2; no voters: e3fefd4dca2d41678187047f06c10187
I20240501 08:12:17.751211 16473 raft_consensus.cc:2781] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:17.751315 16473 raft_consensus.cc:482] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:17.751252 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d21bf5577f4b449a691ef6234376d2" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:17.751408 16473 raft_consensus.cc:3037] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.751470 12972 raft_consensus.cc:2454] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:17.752734 16248 ts_tablet_manager.cc:1888] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.752887 16248 log.cc:1196] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/9e39ce53fb40456285a767ac836fae9d
I20240501 08:12:17.753376 16248 ts_tablet_manager.cc:1909] T 9e39ce53fb40456285a767ac836fae9d P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:17.754340 16718 tablet_bootstrap.cc:492] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:17.754503 16718 ts_tablet_manager.cc:1367] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.156s user 0.003s sys 0.001s
I20240501 08:12:17.754590 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 9e39ce53fb40456285a767ac836fae9d (table rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]) successfully deleted
I20240501 08:12:17.754611 16473 raft_consensus.cc:504] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:17.755322 16718 raft_consensus.cc:348] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:17.755533 16718 raft_consensus.cc:374] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.755652 16718 raft_consensus.cc:729] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.755776 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc25959b69a41c98d196ed88117b26f" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:17.756112 16093 leader_election.cc:302] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 76d830e199eb461c8ca689c9460ee0c2; no voters: 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:17.756700 16473 leader_election.cc:288] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:17.757148 16473 raft_consensus.cc:2781] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:17.757484 16473 raft_consensus.cc:482] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:17.757650 16720 tablet_bootstrap.cc:492] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:17.757781 16473 raft_consensus.cc:3037] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.757819 16720 ts_tablet_manager.cc:1367] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.156s user 0.002s sys 0.003s
I20240501 08:12:17.759897 16498 ts_tablet_manager.cc:1888] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.760150 16498 log.cc:1196] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/ec027f1fcf0b4a5a87e56221f3e1e5cd
I20240501 08:12:17.760762 16498 ts_tablet_manager.cc:1909] T ec027f1fcf0b4a5a87e56221f3e1e5cd P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:17.761775 16498 tablet_replica.cc:330] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:17.761906 16710 tablet.cc:1722] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:17.762169 16498 raft_consensus.cc:2229] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.762398 16498 raft_consensus.cc:2258] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.762606 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet ec027f1fcf0b4a5a87e56221f3e1e5cd (table rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]) successfully deleted
I20240501 08:12:17.763144 16498 ts_tablet_manager.cc:1875] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.763842 16720 raft_consensus.cc:348] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.764345 16718 consensus_queue.cc:260] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:17.764639 13105 raft_consensus.cc:3037] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.766886 16250 ts_tablet_manager.cc:1888] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.767056 16250 log.cc:1196] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/9e39ce53fb40456285a767ac836fae9d
I20240501 08:12:17.767797 13105 raft_consensus.cc:2454] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:17.764083 16720 raft_consensus.cc:374] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.768613 16092 leader_election.cc:302] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:17.768672 16720 raft_consensus.cc:729] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.768958 16720 consensus_queue.cc:260] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.773046 16498 ts_tablet_manager.cc:1888] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.773314 16498 log.cc:1196] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/9e39ce53fb40456285a767ac836fae9d
I20240501 08:12:17.773288 16718 ts_tablet_manager.cc:1398] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240501 08:12:17.773746 16250 ts_tablet_manager.cc:1909] T 9e39ce53fb40456285a767ac836fae9d P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:17.773839 16498 ts_tablet_manager.cc:1909] T 9e39ce53fb40456285a767ac836fae9d P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:17.774252 16473 raft_consensus.cc:504] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:17.775102 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 9e39ce53fb40456285a767ac836fae9d (table rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]) successfully deleted
I20240501 08:12:17.776706 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d21bf5577f4b449a691ef6234376d2" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:17.776942 15756 raft_consensus.cc:3037] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.777280 16720 ts_tablet_manager.cc:1398] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240501 08:12:17.778190 16250 tablet_replica.cc:330] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:17.778326 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 9e39ce53fb40456285a767ac836fae9d (table rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]) successfully deleted
I20240501 08:12:17.778417 16250 raft_consensus.cc:2229] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.778656 16250 raft_consensus.cc:2258] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.779234 16250 ts_tablet_manager.cc:1875] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.779506 16720 tablet_bootstrap.cc:492] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:17.779675 15756 raft_consensus.cc:2454] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:17.780400 16092 leader_election.cc:302] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:17.788326 15872 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 08:12:17.789692 16473 leader_election.cc:288] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:17.789921 16473 raft_consensus.cc:2781] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:17.790041 16473 raft_consensus.cc:686] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:17.790304 16473 consensus_queue.cc:237] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:17.794777 16250 ts_tablet_manager.cc:1888] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:17.794919 16250 log.cc:1196] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/ec027f1fcf0b4a5a87e56221f3e1e5cd
I20240501 08:12:17.795297 16250 ts_tablet_manager.cc:1909] T ec027f1fcf0b4a5a87e56221f3e1e5cd P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:17.796277 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet ec027f1fcf0b4a5a87e56221f3e1e5cd (table rebalance_test_table_14 [id=c3bf94d0aef44991ba373437c141baf6]) successfully deleted
I20240501 08:12:17.800552 16473 raft_consensus.cc:2781] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:17.800735 16473 raft_consensus.cc:686] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:17.801021 16473 consensus_queue.cc:237] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
W20240501 08:12:17.803483 16093 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 47389.6us
I20240501 08:12:17.805140 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d21bf5577f4b449a691ef6234376d2" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:17.805418 12974 raft_consensus.cc:3037] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.805434 13345 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:17.805140 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc25959b69a41c98d196ed88117b26f" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:17.805855 12972 raft_consensus.cc:3037] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:17.808650 12972 raft_consensus.cc:2454] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:17.808699 12974 raft_consensus.cc:2454] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:17.817385 12824 catalog_manager.cc:5484] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:17.818001 12824 catalog_manager.cc:5484] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:17.818329 16720 tablet_bootstrap.cc:654] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.841921 16138 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:17.842453 16720 tablet_bootstrap.cc:492] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:17.842828 16720 ts_tablet_manager.cc:1367] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.005s
I20240501 08:12:17.843768 16720 raft_consensus.cc:348] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:17.844127 16720 raft_consensus.cc:374] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:17.844254 16720 raft_consensus.cc:729] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:17.844651 16720 consensus_queue.cc:260] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:17.846122 16720 ts_tablet_manager.cc:1398] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:17.851330 12828 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:17.851608 12828 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:17.856830 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 7dd45ed878394e30b1f6d5fb7ccff036
I20240501 08:12:17.858148 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dd45ed878394e30b1f6d5fb7ccff036 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:17.859594 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dd45ed878394e30b1f6d5fb7ccff036 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:17.861361 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dd45ed878394e30b1f6d5fb7ccff036 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:17 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:17.882202 15892 raft_consensus.cc:1203] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:17.884399 16753 tablet_replica.cc:330] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:17.884733 16753 raft_consensus.cc:2229] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.885094 16753 raft_consensus.cc:2258] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.891645 16753 ts_tablet_manager.cc:1875] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.907912 16753 ts_tablet_manager.cc:1888] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.909384 16755 tablet_replica.cc:330] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:17.911545 16754 tablet_replica.cc:330] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:17.909623 16755 raft_consensus.cc:2229] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:17.913527 16755 raft_consensus.cc:2258] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.915843 16754 raft_consensus.cc:2229] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:17.916210 16754 raft_consensus.cc:2258] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:17.915906 16005 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:17.922137 16754 ts_tablet_manager.cc:1875] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.924562 13214 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:17.935544 16755 ts_tablet_manager.cc:1875] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:17.959020 16755 ts_tablet_manager.cc:1888] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.960870 15735 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:17.963536 16755 log.cc:1196] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/7dd45ed878394e30b1f6d5fb7ccff036
I20240501 08:12:17.963578 12828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:17.973268 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfb69f10ef84a0ab5e448a301f952f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3574b25c67414c3bb127e96ce77821a5]), partition=
I20240501 08:12:17.973723 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfb69f10ef84a0ab5e448a301f952f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.976908 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfb69f10ef84a0ab5e448a301f952f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3574b25c67414c3bb127e96ce77821a5]), partition=
I20240501 08:12:17.977383 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfb69f10ef84a0ab5e448a301f952f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.982609 16724 tablet_bootstrap.cc:492] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:17.984112 16754 ts_tablet_manager.cc:1888] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:17.984244 16754 log.cc:1196] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/7dd45ed878394e30b1f6d5fb7ccff036
I20240501 08:12:17.984263 16724 tablet_bootstrap.cc:654] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.976730 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 4bfb69f10ef84a0ab5e448a301f952f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3574b25c67414c3bb127e96ce77821a5]), partition=
I20240501 08:12:17.985893 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bfb69f10ef84a0ab5e448a301f952f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:17.986538 16754 ts_tablet_manager.cc:1909] T 7dd45ed878394e30b1f6d5fb7ccff036 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:17.990386 13085 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:17.994360 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 7dd45ed878394e30b1f6d5fb7ccff036 (table rebalancer_dynamic_table_new_name [id=b8a9ac4cc66e40afa62d7f2156c258f4]) successfully deleted
I20240501 08:12:17.994522 16723 tablet_bootstrap.cc:492] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:17.994871 16755 ts_tablet_manager.cc:1909] T 7dd45ed878394e30b1f6d5fb7ccff036 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:17.996371 16723 tablet_bootstrap.cc:654] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:17.997709 16644 raft_consensus.cc:482] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:17.997876 16644 raft_consensus.cc:504] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:17.998704 16644 leader_election.cc:288] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:17.999897 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6856858a304c4e208188897fe2ebdb33" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:17.999948 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6856858a304c4e208188897fe2ebdb33" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:18.000159 15756 raft_consensus.cc:2454] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:18.000169 16024 raft_consensus.cc:2454] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:18.000252 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 7dd45ed878394e30b1f6d5fb7ccff036 (table rebalancer_dynamic_table_new_name [id=b8a9ac4cc66e40afa62d7f2156c258f4]) successfully deleted
I20240501 08:12:18.001199 15825 leader_election.cc:302] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:18.001860 16720 tablet_bootstrap.cc:492] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:18.002488 16644 raft_consensus.cc:2781] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.002591 16644 raft_consensus.cc:482] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.002660 16644 raft_consensus.cc:3037] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.003392 16720 tablet_bootstrap.cc:654] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.004005 16724 tablet_bootstrap.cc:492] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:18.004168 16724 ts_tablet_manager.cc:1367] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20240501 08:12:18.005010 16724 raft_consensus.cc:348] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.005342 16724 raft_consensus.cc:374] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.005470 16724 raft_consensus.cc:729] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.005816 16724 consensus_queue.cc:260] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.007396 16724 ts_tablet_manager.cc:1398] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:18.013976 16644 raft_consensus.cc:504] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:18.014744 16644 leader_election.cc:288] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:18.029795 16753 log.cc:1196] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/7dd45ed878394e30b1f6d5fb7ccff036
I20240501 08:12:18.030474 16753 ts_tablet_manager.cc:1909] T 7dd45ed878394e30b1f6d5fb7ccff036 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:18.031734 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 7dd45ed878394e30b1f6d5fb7ccff036 (table rebalancer_dynamic_table_new_name [id=b8a9ac4cc66e40afa62d7f2156c258f4]) successfully deleted
I20240501 08:12:18.041347 16720 tablet_bootstrap.cc:492] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:18.041663 16720 ts_tablet_manager.cc:1367] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.003s
I20240501 08:12:18.042691 16720 raft_consensus.cc:348] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.043025 16720 raft_consensus.cc:374] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.043150 16720 raft_consensus.cc:729] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.043468 16720 consensus_queue.cc:260] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.044870 16720 ts_tablet_manager.cc:1398] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:18.047547 12953 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:18.047636 16723 tablet_bootstrap.cc:492] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:18.047806 16723 ts_tablet_manager.cc:1367] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.006s
W20240501 08:12:18.048125 15825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 46956.1us
I20240501 08:12:18.049355 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6856858a304c4e208188897fe2ebdb33" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:18.049592 15756 raft_consensus.cc:3037] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.051208 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6856858a304c4e208188897fe2ebdb33" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:18.051417 16024 raft_consensus.cc:3037] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.054148 16024 raft_consensus.cc:2454] T 6856858a304c4e208188897fe2ebdb33 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:18.054580 15756 raft_consensus.cc:2454] T 6856858a304c4e208188897fe2ebdb33 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:18.055115 15825 leader_election.cc:302] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:18.055809 16764 raft_consensus.cc:2781] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.056097 16764 raft_consensus.cc:686] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:18.056502 16764 consensus_queue.cc:237] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:18.065198 16723 raft_consensus.cc:348] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.065459 16723 raft_consensus.cc:374] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.065553 16723 raft_consensus.cc:729] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.065806 16723 consensus_queue.cc:260] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.071581 16723 ts_tablet_manager.cc:1398] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240501 08:12:18.073437 12828 catalog_manager.cc:5484] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:18.183877 16745 consensus_queue.cc:1035] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.197753 12828 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 08:12:18.198101 12828 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240501 08:12:18.203862 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 47ee02938eee4f1ba72279b6e0b6774c
I20240501 08:12:18.204869 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ee02938eee4f1ba72279b6e0b6774c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:18.205229 16248 tablet_replica.cc:330] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:18.205436 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ee02938eee4f1ba72279b6e0b6774c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:18.205523 16248 raft_consensus.cc:2229] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.205868 16250 tablet_replica.cc:330] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:18.205860 16248 raft_consensus.cc:2258] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.206112 16250 raft_consensus.cc:2229] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.206384 16250 raft_consensus.cc:2258] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.206890 16248 ts_tablet_manager.cc:1875] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.207234 16250 ts_tablet_manager.cc:1875] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.209975 16745 consensus_queue.cc:1035] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.213193 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet 47ee02938eee4f1ba72279b6e0b6774c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:18.213506 16251 tablet_replica.cc:330] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:18.213733 16251 raft_consensus.cc:2229] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:18.214090 16251 raft_consensus.cc:2258] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.214855 16251 ts_tablet_manager.cc:1875] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.216699 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet b568f8f64d1b4718b49e4fb55d8a0d26
I20240501 08:12:18.217737 13085 tablet_service.cc:1514] Processing DeleteTablet for tablet b568f8f64d1b4718b49e4fb55d8a0d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:18.218711 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet b568f8f64d1b4718b49e4fb55d8a0d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:18.218724 12828 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet dfdd065cc96c4ec3a248b0ae40a7fedd
I20240501 08:12:18.219039 16498 tablet_replica.cc:330] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:18.219326 16498 raft_consensus.cc:2229] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:18.219833 16498 raft_consensus.cc:2258] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.220606 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet dfdd065cc96c4ec3a248b0ae40a7fedd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:18.220783 16498 ts_tablet_manager.cc:1875] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.224741 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet dfdd065cc96c4ec3a248b0ae40a7fedd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:18.230746 16250 ts_tablet_manager.cc:1888] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.231199 16251 ts_tablet_manager.cc:1888] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.232010 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet b568f8f64d1b4718b49e4fb55d8a0d26 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:18.232084 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet dfdd065cc96c4ec3a248b0ae40a7fedd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:18.235874 16250 log.cc:1196] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/47ee02938eee4f1ba72279b6e0b6774c
I20240501 08:12:18.238834 16250 ts_tablet_manager.cc:1909] T 47ee02938eee4f1ba72279b6e0b6774c P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:18.239665 16248 ts_tablet_manager.cc:1888] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.240839 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 47ee02938eee4f1ba72279b6e0b6774c (table rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]) successfully deleted
I20240501 08:12:18.241214 16498 ts_tablet_manager.cc:1888] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.241355 16498 log.cc:1196] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/b568f8f64d1b4718b49e4fb55d8a0d26
I20240501 08:12:18.241844 16498 ts_tablet_manager.cc:1909] T b568f8f64d1b4718b49e4fb55d8a0d26 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:18.243396 16250 tablet_replica.cc:330] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:18.231333 16251 log.cc:1196] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/47ee02938eee4f1ba72279b6e0b6774c
I20240501 08:12:18.244161 16250 raft_consensus.cc:2229] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.244565 16251 ts_tablet_manager.cc:1909] T 47ee02938eee4f1ba72279b6e0b6774c P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:18.244972 16250 raft_consensus.cc:2258] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.246707 16250 ts_tablet_manager.cc:1875] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.247460 16251 tablet_replica.cc:330] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:18.247730 16251 raft_consensus.cc:2229] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.247720 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 47ee02938eee4f1ba72279b6e0b6774c (table rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]) successfully deleted
I20240501 08:12:18.247954 16251 raft_consensus.cc:2258] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.248685 16251 ts_tablet_manager.cc:1875] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.251884 16248 log.cc:1196] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/47ee02938eee4f1ba72279b6e0b6774c
I20240501 08:12:18.252420 16248 ts_tablet_manager.cc:1909] T 47ee02938eee4f1ba72279b6e0b6774c P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:18.253948 16498 tablet_replica.cc:330] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:18.254181 16498 raft_consensus.cc:2229] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:18.255012 16248 tablet_replica.cc:330] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:18.255003 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet b568f8f64d1b4718b49e4fb55d8a0d26 (table rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]) successfully deleted
I20240501 08:12:18.255223 16248 raft_consensus.cc:2229] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.255254 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 47ee02938eee4f1ba72279b6e0b6774c (table rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]) successfully deleted
I20240501 08:12:18.255470 16248 raft_consensus.cc:2258] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.255865 16498 raft_consensus.cc:2258] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.256994 16248 ts_tablet_manager.cc:1875] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.257532 16498 ts_tablet_manager.cc:1875] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.262327 16250 ts_tablet_manager.cc:1888] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.262457 16250 log.cc:1196] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/b568f8f64d1b4718b49e4fb55d8a0d26
I20240501 08:12:18.263056 16250 ts_tablet_manager.cc:1909] T b568f8f64d1b4718b49e4fb55d8a0d26 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:18.265184 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet b568f8f64d1b4718b49e4fb55d8a0d26 (table rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]) successfully deleted
I20240501 08:12:18.270486 16248 ts_tablet_manager.cc:1888] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.270695 16248 log.cc:1196] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/b568f8f64d1b4718b49e4fb55d8a0d26
I20240501 08:12:18.271195 16248 ts_tablet_manager.cc:1909] T b568f8f64d1b4718b49e4fb55d8a0d26 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:18.271958 16251 ts_tablet_manager.cc:1888] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.272162 16251 log.cc:1196] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/dfdd065cc96c4ec3a248b0ae40a7fedd
I20240501 08:12:18.272648 16251 ts_tablet_manager.cc:1909] T dfdd065cc96c4ec3a248b0ae40a7fedd P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:18.272716 16498 ts_tablet_manager.cc:1888] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.272920 16498 log.cc:1196] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/dfdd065cc96c4ec3a248b0ae40a7fedd
I20240501 08:12:18.273350 16498 ts_tablet_manager.cc:1909] T dfdd065cc96c4ec3a248b0ae40a7fedd P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:18.273829 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet dfdd065cc96c4ec3a248b0ae40a7fedd (table rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]) successfully deleted
I20240501 08:12:18.274166 16248 tablet_replica.cc:330] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:18.274709 16248 raft_consensus.cc:2229] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.274786 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet dfdd065cc96c4ec3a248b0ae40a7fedd (table rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]) successfully deleted
I20240501 08:12:18.275033 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet b568f8f64d1b4718b49e4fb55d8a0d26 (table rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]) successfully deleted
I20240501 08:12:18.275454 16248 raft_consensus.cc:2258] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.276072 16248 ts_tablet_manager.cc:1875] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.282157 16248 ts_tablet_manager.cc:1888] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.282306 16248 log.cc:1196] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/dfdd065cc96c4ec3a248b0ae40a7fedd
I20240501 08:12:18.282770 16248 ts_tablet_manager.cc:1909] T dfdd065cc96c4ec3a248b0ae40a7fedd P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:18.283844 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet dfdd065cc96c4ec3a248b0ae40a7fedd (table rebalance_test_table_16 [id=2ceb56f187844e50969e5632fde934fc]) successfully deleted
I20240501 08:12:18.309414 13235 consensus_queue.cc:237] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:18.312377 12974 raft_consensus.cc:1261] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:18.313287 16224 consensus_queue.cc:1035] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:18.321003 13105 raft_consensus.cc:1261] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:18.321584 16224 consensus_queue.cc:1035] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:18.324335 16224 raft_consensus.cc:2932] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:18.325054 13105 raft_consensus.cc:2932] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
W20240501 08:12:18.329284 13170 consensus_peers.cc:485] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4fc5dbfe3a9b4c84a3217981527e73ea. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:18.330248 12828 catalog_manager.cc:5484] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:18.332872 12974 raft_consensus.cc:2932] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:18.355249 16473 consensus_queue.cc:1035] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.360791 16473 consensus_queue.cc:1035] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.375339 12974 consensus_queue.cc:237] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:18.377792 13235 raft_consensus.cc:1261] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:18.377897 13364 raft_consensus.cc:1261] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:18.378443 16735 consensus_queue.cc:1035] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:18.378757 12909 consensus_peers.cc:485] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd9a5b5333d54f6d9d8186e4d9b1b1be. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:18.378818 16813 consensus_queue.cc:1035] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:18.381362 16813 raft_consensus.cc:2932] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:18.381974 13235 raft_consensus.cc:2932] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:18.384935 13364 raft_consensus.cc:2932] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:18.391124 12824 catalog_manager.cc:5484] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:18.402359 16367 raft_consensus.cc:482] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.402570 16367 raft_consensus.cc:504] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.403378 16367 leader_election.cc:288] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:18.403904 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfb69f10ef84a0ab5e448a301f952f5" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:18.404114 15892 raft_consensus.cc:2454] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:18.404263 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfb69f10ef84a0ab5e448a301f952f5" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:18.404515 15756 raft_consensus.cc:2454] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:18.404541 15957 leader_election.cc:302] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:18.404775 16367 raft_consensus.cc:2781] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.404857 16367 raft_consensus.cc:482] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.404920 16367 raft_consensus.cc:3037] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.407613 16367 raft_consensus.cc:504] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.408746 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfb69f10ef84a0ab5e448a301f952f5" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:18.408818 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bfb69f10ef84a0ab5e448a301f952f5" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:18.408977 15756 raft_consensus.cc:3037] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.409054 15892 raft_consensus.cc:3037] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.411520 15756 raft_consensus.cc:2454] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:18.411799 15892 raft_consensus.cc:2454] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:18.412025 15959 leader_election.cc:302] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:18.412385 16738 raft_consensus.cc:2781] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.412411 16367 leader_election.cc:288] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:18.412802 16738 raft_consensus.cc:686] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:18.413133 16738 consensus_queue.cc:237] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.423457 12828 catalog_manager.cc:5484] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:18.431114 13105 consensus_queue.cc:237] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:18.434707 13364 raft_consensus.cc:1261] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:18.435034 13235 raft_consensus.cc:1261] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:18.435364 16074 consensus_queue.cc:1035] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:18.435907 16074 consensus_queue.cc:1035] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:18.438850 16074 raft_consensus.cc:2932] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:18.439457 13235 raft_consensus.cc:2932] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:18.445349 12828 catalog_manager.cc:5484] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
W20240501 08:12:18.447772 13038 consensus_peers.cc:485] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5179549829d451eb3a6e2083ef59b7f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:18.451189 13364 raft_consensus.cc:2932] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:18.470685 16764 consensus_queue.cc:1035] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.488404 16764 consensus_queue.cc:1035] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.593283 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:18.600013 15892 raft_consensus.cc:1261] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:18.600162 15756 raft_consensus.cc:1261] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:18.600732 16367 consensus_queue.cc:1035] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.601150 16826 consensus_queue.cc:1035] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:18.602253 12825 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:18.625530 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 181a5dd80e5a448b9a63d037e8ae93e7 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.625515 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 181a5dd80e5a448b9a63d037e8ae93e7 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.626009 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181a5dd80e5a448b9a63d037e8ae93e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.626011 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181a5dd80e5a448b9a63d037e8ae93e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.627136 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 04ecd52892254c99857cd1d89f514f99 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.627600 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ecd52892254c99857cd1d89f514f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.627687 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 29335562faf0492f9acf2848fc582dd8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.628129 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29335562faf0492f9acf2848fc582dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.629001 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 29335562faf0492f9acf2848fc582dd8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.629200 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 04ecd52892254c99857cd1d89f514f99 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.629455 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29335562faf0492f9acf2848fc582dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.629712 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ecd52892254c99857cd1d89f514f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.630091 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 181a5dd80e5a448b9a63d037e8ae93e7 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.630493 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181a5dd80e5a448b9a63d037e8ae93e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.633009 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 29335562faf0492f9acf2848fc582dd8 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.633383 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29335562faf0492f9acf2848fc582dd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.639103 12832 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 4bfb69f10ef84a0ab5e448a301f952f5 (table rebalancer_dynamic_table [id=3574b25c67414c3bb127e96ce77821a5]). Expected version 1 got 0
I20240501 08:12:18.645972 16877 tablet_bootstrap.cc:492] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:18.647181 16876 tablet_bootstrap.cc:492] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:18.649220 16877 tablet_bootstrap.cc:654] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.649658 16876 tablet_bootstrap.cc:654] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.653611 16880 tablet_bootstrap.cc:492] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:18.654523 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 04ecd52892254c99857cd1d89f514f99 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:18.655061 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ecd52892254c99857cd1d89f514f99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.661294 16876 tablet_bootstrap.cc:492] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:18.661741 16880 tablet_bootstrap.cc:654] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.663300 16876 ts_tablet_manager.cc:1367] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20240501 08:12:18.663556 16877 tablet_bootstrap.cc:492] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:18.663797 16877 ts_tablet_manager.cc:1367] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20240501 08:12:18.664937 16881 tablet_bootstrap.cc:492] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:18.666013 16876 raft_consensus.cc:348] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:18.666146 16877 raft_consensus.cc:348] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.666666 16877 raft_consensus.cc:374] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.666836 16877 raft_consensus.cc:729] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.668274 16880 tablet_bootstrap.cc:492] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:18.668707 16877 consensus_queue.cc:260] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.669487 16883 tablet_bootstrap.cc:492] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:18.670387 16877 ts_tablet_manager.cc:1398] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 08:12:18.675238 16881 tablet_bootstrap.cc:654] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.675784 16876 raft_consensus.cc:374] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.675945 16876 raft_consensus.cc:729] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.676229 16876 consensus_queue.cc:260] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:18.677248 16880 ts_tablet_manager.cc:1367] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20240501 08:12:18.678122 16880 raft_consensus.cc:348] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.678350 16880 raft_consensus.cc:374] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.678449 16880 raft_consensus.cc:729] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.678694 16880 consensus_queue.cc:260] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.679271 16880 ts_tablet_manager.cc:1398] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:18.681375 16876 ts_tablet_manager.cc:1398] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240501 08:12:18.683138 16884 tablet_bootstrap.cc:492] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:18.686352 16884 tablet_bootstrap.cc:654] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.686767 16883 tablet_bootstrap.cc:654] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.692924 16881 tablet_bootstrap.cc:492] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:18.693516 16881 ts_tablet_manager.cc:1367] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.004s
I20240501 08:12:18.695664 16881 raft_consensus.cc:348] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:18.696521 16881 raft_consensus.cc:374] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.696810 16881 raft_consensus.cc:729] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.697271 16881 consensus_queue.cc:260] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:18.699458 16881 ts_tablet_manager.cc:1398] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240501 08:12:18.703501 16888 tablet_bootstrap.cc:492] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
W20240501 08:12:18.707751 16874 log.cc:924] Time spent T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20240501 08:12:18.707800 12825 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 4bfb69f10ef84a0ab5e448a301f952f5 (table rebalancer_dynamic_table [id=3574b25c67414c3bb127e96ce77821a5]). Expected version 1 got 0
I20240501 08:12:18.709973 16889 tablet.cc:1722] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:18.710748 16889 tablet.cc:1620] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:18.710798 16883 tablet_bootstrap.cc:492] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:18.710951 16883 ts_tablet_manager.cc:1367] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240501 08:12:18.711999 16883 raft_consensus.cc:348] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:18.712392 16883 raft_consensus.cc:374] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.712551 16883 raft_consensus.cc:729] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.712965 16883 consensus_queue.cc:260] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:18.713734 16883 ts_tablet_manager.cc:1398] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:18.715744 16883 tablet_bootstrap.cc:492] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:18.721844 16888 tablet_bootstrap.cc:654] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.724826 16883 tablet_bootstrap.cc:654] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.725585 16892 tablet.cc:1722] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:18.726332 16892 tablet.cc:1620] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:18.725428 16891 tablet.cc:1722] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:18.726994 16891 tablet.cc:1620] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:18.728040 12829 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 08:12:18.728268 12829 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240501 08:12:18.731050 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=3574b25c67414c3bb127e96ce77821a5] alter complete (version 1)
I20240501 08:12:18.732272 12825 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 4bfb69f10ef84a0ab5e448a301f952f5 (table rebalancer_dynamic_table [id=3574b25c67414c3bb127e96ce77821a5]). Expected version 1 got 0
I20240501 08:12:18.736192 12829 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet ad1faf7987004fe29256bdbf993ce2e5
I20240501 08:12:18.737363 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1faf7987004fe29256bdbf993ce2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:18.737381 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1faf7987004fe29256bdbf993ce2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:18.737648 16250 tablet_replica.cc:330] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:18.737674 16248 tablet_replica.cc:330] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:18.737726 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1faf7987004fe29256bdbf993ce2e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:18.737882 16250 raft_consensus.cc:2229] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.737908 16248 raft_consensus.cc:2229] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.738006 16498 tablet_replica.cc:330] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:18.738133 16250 raft_consensus.cc:2258] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.738158 16248 raft_consensus.cc:2258] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.738224 16498 raft_consensus.cc:2229] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:18.738574 16498 raft_consensus.cc:2258] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.738904 16250 ts_tablet_manager.cc:1875] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.738934 16248 ts_tablet_manager.cc:1875] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.739217 16498 ts_tablet_manager.cc:1875] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.745380 12829 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet bf1baafa5548431180ca3bc8d35607ad
I20240501 08:12:18.746191 16498 ts_tablet_manager.cc:1888] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.748644 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1baafa5548431180ca3bc8d35607ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:18.748620 12829 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet d7cd8edcdc5c4f22b5ba33224887d027
I20240501 08:12:18.749320 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1baafa5548431180ca3bc8d35607ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:18.749621 16251 tablet_replica.cc:330] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:18.749838 16251 raft_consensus.cc:2229] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.750109 16251 raft_consensus.cc:2258] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.750344 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet d7cd8edcdc5c4f22b5ba33224887d027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:18.750824 16251 ts_tablet_manager.cc:1875] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.751199 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet d7cd8edcdc5c4f22b5ba33224887d027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:18.749094 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet bf1baafa5548431180ca3bc8d35607ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:18.749310 13215 tablet_service.cc:1514] Processing DeleteTablet for tablet d7cd8edcdc5c4f22b5ba33224887d027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:18 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:18.751798 16498 log.cc:1196] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/ad1faf7987004fe29256bdbf993ce2e5
I20240501 08:12:18.752342 16498 ts_tablet_manager.cc:1909] T ad1faf7987004fe29256bdbf993ce2e5 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:18.753371 16498 tablet_replica.cc:330] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:18.753613 16498 raft_consensus.cc:2229] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:18.753945 16498 raft_consensus.cc:2258] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.755016 16498 ts_tablet_manager.cc:1875] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.756028 12832 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 4bfb69f10ef84a0ab5e448a301f952f5 (table rebalancer_dynamic_table [id=3574b25c67414c3bb127e96ce77821a5]). Expected version 1 got 0
I20240501 08:12:18.756237 16250 ts_tablet_manager.cc:1888] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.757158 16250 log.cc:1196] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/ad1faf7987004fe29256bdbf993ce2e5
I20240501 08:12:18.757279 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet ad1faf7987004fe29256bdbf993ce2e5 (table rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]) successfully deleted
I20240501 08:12:18.758882 16250 ts_tablet_manager.cc:1909] T ad1faf7987004fe29256bdbf993ce2e5 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:18.760818 16248 ts_tablet_manager.cc:1888] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.762435 16251 ts_tablet_manager.cc:1888] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.763630 16251 log.cc:1196] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/bf1baafa5548431180ca3bc8d35607ad
I20240501 08:12:18.764350 16251 ts_tablet_manager.cc:1909] T bf1baafa5548431180ca3bc8d35607ad P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:18.764480 16498 ts_tablet_manager.cc:1888] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.764652 16498 log.cc:1196] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/bf1baafa5548431180ca3bc8d35607ad
I20240501 08:12:18.764880 16248 log.cc:1196] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad1faf7987004fe29256bdbf993ce2e5
I20240501 08:12:18.765120 16498 ts_tablet_manager.cc:1909] T bf1baafa5548431180ca3bc8d35607ad P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:18.765419 16884 tablet_bootstrap.cc:492] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:18.765590 16884 ts_tablet_manager.cc:1367] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.083s user 0.005s sys 0.010s
I20240501 08:12:18.766270 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet ad1faf7987004fe29256bdbf993ce2e5 (table rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]) successfully deleted
I20240501 08:12:18.766270 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet bf1baafa5548431180ca3bc8d35607ad (table rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]) successfully deleted
I20240501 08:12:18.766431 16884 raft_consensus.cc:348] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.766681 16250 tablet_replica.cc:330] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:18.766896 16250 raft_consensus.cc:2229] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:18.766657 16884 raft_consensus.cc:374] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.768007 16884 raft_consensus.cc:729] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.768326 16248 ts_tablet_manager.cc:1909] T ad1faf7987004fe29256bdbf993ce2e5 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:18.768329 16884 consensus_queue.cc:260] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.769150 16250 raft_consensus.cc:2258] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.769891 16250 ts_tablet_manager.cc:1875] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.771587 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet bf1baafa5548431180ca3bc8d35607ad (table rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]) successfully deleted
I20240501 08:12:18.772267 16884 ts_tablet_manager.cc:1398] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240501 08:12:18.772486 16498 tablet_replica.cc:330] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:18.773131 16498 raft_consensus.cc:2229] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.773516 16884 tablet_bootstrap.cc:492] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:18.773703 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet ad1faf7987004fe29256bdbf993ce2e5 (table rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]) successfully deleted
I20240501 08:12:18.773762 16498 raft_consensus.cc:2258] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.774562 16498 ts_tablet_manager.cc:1875] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.775424 16884 tablet_bootstrap.cc:654] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:18.777405 16248 tablet_replica.cc:330] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:18.777626 16248 raft_consensus.cc:2229] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.781627 16248 raft_consensus.cc:2258] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.782415 16248 ts_tablet_manager.cc:1875] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.784781 16888 tablet_bootstrap.cc:492] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:18.784958 16888 ts_tablet_manager.cc:1367] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.082s user 0.001s sys 0.007s
I20240501 08:12:18.786016 16888 raft_consensus.cc:348] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.786314 16888 raft_consensus.cc:374] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.786415 16888 raft_consensus.cc:729] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.786603 16250 ts_tablet_manager.cc:1888] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.786739 16250 log.cc:1196] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/d7cd8edcdc5c4f22b5ba33224887d027
I20240501 08:12:18.786650 16888 consensus_queue.cc:260] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.787307 16250 ts_tablet_manager.cc:1909] T d7cd8edcdc5c4f22b5ba33224887d027 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:18.787405 16896 ts_tablet_manager.cc:902] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:18.789263 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet d7cd8edcdc5c4f22b5ba33224887d027 (table rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]) successfully deleted
I20240501 08:12:18.791258 16498 ts_tablet_manager.cc:1888] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.791343 16883 tablet_bootstrap.cc:492] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:18.791412 16498 log.cc:1196] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/d7cd8edcdc5c4f22b5ba33224887d027
I20240501 08:12:18.791515 16883 ts_tablet_manager.cc:1367] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.003s
I20240501 08:12:18.791855 16498 ts_tablet_manager.cc:1909] T d7cd8edcdc5c4f22b5ba33224887d027 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:18.792335 16884 tablet_bootstrap.cc:492] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:18.792403 16883 raft_consensus.cc:348] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.792574 16884 ts_tablet_manager.cc:1367] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20240501 08:12:18.792641 16883 raft_consensus.cc:374] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.792959 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet d7cd8edcdc5c4f22b5ba33224887d027 (table rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]) successfully deleted
I20240501 08:12:18.793613 16884 raft_consensus.cc:348] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.793907 16884 raft_consensus.cc:374] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:18.794312 16883 raft_consensus.cc:729] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.794389 16884 raft_consensus.cc:729] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:18.794735 16884 consensus_queue.cc:260] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.795413 16884 ts_tablet_manager.cc:1398] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:18.798766 16883 consensus_queue.cc:260] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.799633 16883 ts_tablet_manager.cc:1398] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20240501 08:12:18.800393 16888 ts_tablet_manager.cc:1398] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240501 08:12:18.804530 16764 raft_consensus.cc:482] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.804739 16764 raft_consensus.cc:504] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.805487 16764 leader_election.cc:288] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:18.806295 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29335562faf0492f9acf2848fc582dd8" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:18.806537 13105 raft_consensus.cc:2454] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:18.807332 16248 ts_tablet_manager.cc:1888] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.807410 15825 leader_election.cc:302] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:18.807694 16248 log.cc:1196] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/bf1baafa5548431180ca3bc8d35607ad
I20240501 08:12:18.808035 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29335562faf0492f9acf2848fc582dd8" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:18.808332 16157 raft_consensus.cc:2454] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:18.808444 16248 ts_tablet_manager.cc:1909] T bf1baafa5548431180ca3bc8d35607ad P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:18.810015 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet bf1baafa5548431180ca3bc8d35607ad (table rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]) successfully deleted
I20240501 08:12:18.810612 16248 tablet_replica.cc:330] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:18.810971 16248 raft_consensus.cc:2229] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:18.811300 16248 raft_consensus.cc:2258] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:18.812011 16764 raft_consensus.cc:2781] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.812119 16764 raft_consensus.cc:482] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.812232 16764 raft_consensus.cc:3037] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.812408 16248 ts_tablet_manager.cc:1875] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:18.814272 16764 raft_consensus.cc:504] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.815340 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29335562faf0492f9acf2848fc582dd8" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:18.815596 16157 raft_consensus.cc:3037] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.815718 16764 leader_election.cc:288] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:18.816514 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29335562faf0492f9acf2848fc582dd8" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:18.816721 13104 raft_consensus.cc:3037] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.816826 16896 tablet_copy_client.cc:308] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:12:18.819011 13104 raft_consensus.cc:2454] T 29335562faf0492f9acf2848fc582dd8 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:18.819487 15825 leader_election.cc:302] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:18.819836 16764 raft_consensus.cc:2781] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.820468 16764 raft_consensus.cc:686] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:18.820756 16764 consensus_queue.cc:237] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:18.822857 16248 ts_tablet_manager.cc:1888] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240501 08:12:18.823113 16248 log.cc:1196] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/d7cd8edcdc5c4f22b5ba33224887d027
I20240501 08:12:18.823352 16157 raft_consensus.cc:2454] T 29335562faf0492f9acf2848fc582dd8 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:18.823841 16248 ts_tablet_manager.cc:1909] T d7cd8edcdc5c4f22b5ba33224887d027 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:18.827375 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet d7cd8edcdc5c4f22b5ba33224887d027 (table rebalance_test_table_18 [id=dc6dee4abd8f47a9bf045445762a8869]) successfully deleted
I20240501 08:12:18.836669 12828 catalog_manager.cc:5484] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:18.837289 12829 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:18.837523 13255 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet 4fc5dbfe3a9b4c84a3217981527e73ea from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:37937)
I20240501 08:12:18.837746 13255 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet 4fc5dbfe3a9b4c84a3217981527e73ea from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:37937: session id = 5dd722c770d34d6c9154a87917117161-4fc5dbfe3a9b4c84a3217981527e73ea
I20240501 08:12:18.841162 13255 tablet_copy_source_session.cc:205] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:18.842747 16379 raft_consensus.cc:482] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:18.842926 16379 raft_consensus.cc:504] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:18.843493 16896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc5dbfe3a9b4c84a3217981527e73ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:18.843756 16379 leader_election.cc:288] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:18.845393 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ecd52892254c99857cd1d89f514f99" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:18.845631 13234 raft_consensus.cc:2454] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:18.846527 16889 tablet.cc:1722] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:18.846756 13301 leader_election.cc:302] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:18.847031 16889 tablet.cc:1620] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:18.847208 16892 tablet.cc:1722] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:18.847276 15502 raft_consensus.cc:2781] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:18.847373 15502 raft_consensus.cc:482] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:18.847476 15502 raft_consensus.cc:3037] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.847674 16892 tablet.cc:1620] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:18.850405 16896 tablet_copy_client.cc:783] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:18.850663 16896 tablet_copy_client.cc:647] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:18.852219 15502 raft_consensus.cc:504] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:18.853143 16896 tablet_copy_client.cc:523] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:18.854146 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ecd52892254c99857cd1d89f514f99" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:18.854378 13235 raft_consensus.cc:3037] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.855969 16896 tablet_bootstrap.cc:492] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:18.856655 15502 leader_election.cc:288] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:18.856774 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ecd52892254c99857cd1d89f514f99" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:18.857023 16158 raft_consensus.cc:3037] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:18.859890 16158 raft_consensus.cc:2454] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:18.860877 16891 tablet.cc:1722] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:18.861390 16891 tablet.cc:1620] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:18.862829 13298 leader_election.cc:302] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:18.863202 15502 raft_consensus.cc:2781] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:18.863898 13235 raft_consensus.cc:2454] T 04ecd52892254c99857cd1d89f514f99 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:18.863940 15502 raft_consensus.cc:686] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:18.864234 15502 consensus_queue.cc:237] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:18.873463 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ecd52892254c99857cd1d89f514f99" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:18.877846 16157 raft_consensus.cc:2362] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1. Re-sending same reply.
I20240501 08:12:18.880216 12828 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=3574b25c67414c3bb127e96ce77821a5] alter complete (version 2)
I20240501 08:12:18.887764 16909 ts_tablet_manager.cc:902] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:18.940156 15872 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:12:18.952783 12829 catalog_manager.cc:5484] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:18.976655 16138 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:18.984531 12829 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:18.999191 13345 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:19.011013 16889 tablet.cc:1722] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:19.018432 16891 tablet.cc:1722] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:19.023782 16892 tablet.cc:1722] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:19.032768 16005 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:12:19.038713 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=3574b25c67414c3bb127e96ce77821a5] alter complete (version 3)
I20240501 08:12:19.049378 16927 ts_tablet_manager.cc:902] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:19.049350 16909 tablet_copy_client.cc:308] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:12:19.050158 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet cd9a5b5333d54f6d9d8186e4d9b1b1be from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:47087)
I20240501 08:12:19.050328 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet cd9a5b5333d54f6d9d8186e4d9b1b1be from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:47087: session id = 982f49fc87f149439cda0de2261b6bfd-cd9a5b5333d54f6d9d8186e4d9b1b1be
I20240501 08:12:19.053375 12994 tablet_copy_source_session.cc:205] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:19.054981 16909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd9a5b5333d54f6d9d8186e4d9b1b1be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.062800 13215 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:19.065017 16495 raft_consensus.cc:482] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.065227 16495 raft_consensus.cc:504] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:19.066133 16495 leader_election.cc:288] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:19.067436 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181a5dd80e5a448b9a63d037e8ae93e7" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:19.067723 16024 raft_consensus.cc:2454] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:19.078186 16909 tablet_copy_client.cc:783] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:19.078817 16909 tablet_copy_client.cc:647] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:19.081558 16909 tablet_copy_client.cc:523] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:19.087409 15691 leader_election.cc:302] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:19.087896 16495 raft_consensus.cc:2781] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.088025 16495 raft_consensus.cc:482] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.088136 16495 raft_consensus.cc:3037] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.091061 16495 raft_consensus.cc:504] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:19.092133 16495 leader_election.cc:288] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:19.093109 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181a5dd80e5a448b9a63d037e8ae93e7" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:19.093364 16024 raft_consensus.cc:3037] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.096387 16024 raft_consensus.cc:2454] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:19.097384 15691 leader_election.cc:302] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:19.097512 15735 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 08:12:19.099123 16495 raft_consensus.cc:2781] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.103036 12954 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:19.121188 16927 tablet_copy_client.cc:308] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:12:19.123430 13125 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet d5179549829d451eb3a6e2083ef59b7f from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:47641)
I20240501 08:12:19.123651 13125 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet d5179549829d451eb3a6e2083ef59b7f from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:47641: session id = 4d516916eeac468db514eeb928f06190-d5179549829d451eb3a6e2083ef59b7f
I20240501 08:12:19.126668 13125 tablet_copy_source_session.cc:205] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:19.129067 16927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5179549829d451eb3a6e2083ef59b7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.136013 16495 raft_consensus.cc:686] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:19.136518 16495 consensus_queue.cc:237] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:19.138178 16927 tablet_copy_client.cc:783] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:19.138456 16927 tablet_copy_client.cc:647] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:19.140887 12829 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:19.141148 12829 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:19.142208 16909 tablet_bootstrap.cc:492] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:19.142820 13084 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:19.145607 12829 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 4bfb69f10ef84a0ab5e448a301f952f5
I20240501 08:12:19.146919 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfb69f10ef84a0ab5e448a301f952f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:19 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:19.147138 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfb69f10ef84a0ab5e448a301f952f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:19 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:19.158630 16940 tablet_replica.cc:330] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:19.159104 16940 raft_consensus.cc:2229] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:19.160849 16927 tablet_copy_client.cc:523] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:19.161517 16940 raft_consensus.cc:2258] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:19.162652 16940 ts_tablet_manager.cc:1875] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:19.147344 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bfb69f10ef84a0ab5e448a301f952f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:19 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:19.174751 12829 catalog_manager.cc:5484] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:19.186529 16941 tablet_replica.cc:330] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:19.186803 16941 raft_consensus.cc:2229] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:19.187045 16941 raft_consensus.cc:2258] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:19.193482 16941 ts_tablet_manager.cc:1875] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:19.195266 16840 raft_consensus.cc:482] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.195448 16840 raft_consensus.cc:504] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:19.196586 16840 leader_election.cc:288] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:19.197477 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181a5dd80e5a448b9a63d037e8ae93e7" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:19.197772 16024 raft_consensus.cc:2379] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3fefd4dca2d41678187047f06c10187 in current term 1: Already voted for candidate 982f49fc87f149439cda0de2261b6bfd in this term.
I20240501 08:12:19.207680 16896 tablet_bootstrap.cc:492] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:19.208180 16896 tablet_bootstrap.cc:492] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:12:19.208333 16927 tablet_bootstrap.cc:492] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:19.208514 16896 ts_tablet_manager.cc:1367] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.353s user 0.061s sys 0.023s
I20240501 08:12:19.209334 16896 raft_consensus.cc:348] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.209584 16896 raft_consensus.cc:729] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.209833 16896 consensus_queue.cc:260] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.210429 16896 ts_tablet_manager.cc:1398] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:19.212036 13255 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-4fc5dbfe3a9b4c84a3217981527e73ea received from {username='slave'} at 127.6.56.71:37937
I20240501 08:12:19.212260 13255 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 5dd722c770d34d6c9154a87917117161-4fc5dbfe3a9b4c84a3217981527e73ea on tablet 4fc5dbfe3a9b4c84a3217981527e73ea with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:12:19.215124 16940 ts_tablet_manager.cc:1888] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:19.215241 16940 log.cc:1196] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/4bfb69f10ef84a0ab5e448a301f952f5
I20240501 08:12:19.215664 16940 ts_tablet_manager.cc:1909] T 4bfb69f10ef84a0ab5e448a301f952f5 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:19.221729 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181a5dd80e5a448b9a63d037e8ae93e7" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:19.222613 13040 leader_election.cc:302] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3fefd4dca2d41678187047f06c10187; no voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:19.223397 16840 raft_consensus.cc:3037] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.226339 16840 raft_consensus.cc:2726] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:19.227421 16941 ts_tablet_manager.cc:1888] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:19.227552 16941 log.cc:1196] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/4bfb69f10ef84a0ab5e448a301f952f5
I20240501 08:12:19.228057 16941 ts_tablet_manager.cc:1909] T 4bfb69f10ef84a0ab5e448a301f952f5 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:19.230283 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 4bfb69f10ef84a0ab5e448a301f952f5 (table rebalancer_dynamic_table_new_name [id=3574b25c67414c3bb127e96ce77821a5]) successfully deleted
I20240501 08:12:19.233014 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 4bfb69f10ef84a0ab5e448a301f952f5 (table rebalancer_dynamic_table_new_name [id=3574b25c67414c3bb127e96ce77821a5]) successfully deleted
I20240501 08:12:19.248083 12829 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:19.250100 16942 tablet_replica.cc:330] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:19.250337 16942 raft_consensus.cc:2229] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:19.250279 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181a5dd80e5a448b9a63d037e8ae93e7" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:19.250591 16942 raft_consensus.cc:2258] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:19.254386 13104 raft_consensus.cc:2454] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:19.254844 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "181a5dd80e5a448b9a63d037e8ae93e7" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:19.255585 13105 raft_consensus.cc:2362] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1. Re-sending same reply.
I20240501 08:12:19.256690 16942 ts_tablet_manager.cc:1875] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:19.258492 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 77e38e896cd5491b97cf470080d884b7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=969800c6b9e044e9b164ac3460f4e306]), partition=
I20240501 08:12:19.258704 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 77e38e896cd5491b97cf470080d884b7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=969800c6b9e044e9b164ac3460f4e306]), partition=
I20240501 08:12:19.258858 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e38e896cd5491b97cf470080d884b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.259055 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e38e896cd5491b97cf470080d884b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.263386 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 77e38e896cd5491b97cf470080d884b7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=969800c6b9e044e9b164ac3460f4e306]), partition=
I20240501 08:12:19.263743 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77e38e896cd5491b97cf470080d884b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.266196 16883 tablet_bootstrap.cc:492] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:19.267738 16883 tablet_bootstrap.cc:654] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.274948 16888 tablet_bootstrap.cc:492] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:19.275435 16883 tablet_bootstrap.cc:492] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:19.275583 16883 ts_tablet_manager.cc:1367] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240501 08:12:19.276621 16888 tablet_bootstrap.cc:654] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.276580 16883 raft_consensus.cc:348] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.276795 16883 raft_consensus.cc:374] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.276883 16883 raft_consensus.cc:729] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.277211 16883 consensus_queue.cc:260] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.278607 16883 ts_tablet_manager.cc:1398] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:19.288997 16942 ts_tablet_manager.cc:1888] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:19.289425 16942 log.cc:1196] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/4bfb69f10ef84a0ab5e448a301f952f5
I20240501 08:12:19.289969 16942 ts_tablet_manager.cc:1909] T 4bfb69f10ef84a0ab5e448a301f952f5 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:19.291198 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 4bfb69f10ef84a0ab5e448a301f952f5 (table rebalancer_dynamic_table_new_name [id=3574b25c67414c3bb127e96ce77821a5]) successfully deleted
I20240501 08:12:19.293583 16888 tablet_bootstrap.cc:492] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:19.293751 16888 ts_tablet_manager.cc:1367] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20240501 08:12:19.294476 16888 raft_consensus.cc:348] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.294678 16888 raft_consensus.cc:374] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.294766 16888 raft_consensus.cc:729] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.295049 16888 consensus_queue.cc:260] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.295607 16888 ts_tablet_manager.cc:1398] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:19.302196 16945 tablet_bootstrap.cc:492] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:19.303833 16945 tablet_bootstrap.cc:654] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.305876 16945 tablet_bootstrap.cc:492] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:19.306084 16945 ts_tablet_manager.cc:1367] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:19.306910 16945 raft_consensus.cc:348] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.307077 16945 raft_consensus.cc:374] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.307189 16945 raft_consensus.cc:729] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.307432 16945 consensus_queue.cc:260] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.309021 16945 ts_tablet_manager.cc:1398] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:19.317870 16764 consensus_queue.cc:1035] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.353432 16745 raft_consensus.cc:482] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.353607 16745 raft_consensus.cc:504] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.354408 16745 leader_election.cc:288] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:19.355885 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e38e896cd5491b97cf470080d884b7" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:19.356206 16024 raft_consensus.cc:2454] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:19.356941 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e38e896cd5491b97cf470080d884b7" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:19.357189 15755 raft_consensus.cc:2454] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:19.364511 16092 leader_election.cc:302] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:19.365072 16745 raft_consensus.cc:2781] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.365207 16745 raft_consensus.cc:482] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.365303 16745 raft_consensus.cc:3037] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.368079 16745 raft_consensus.cc:504] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.369745 16745 leader_election.cc:288] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:19.369853 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e38e896cd5491b97cf470080d884b7" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:19.370029 16764 consensus_queue.cc:1035] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.370564 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77e38e896cd5491b97cf470080d884b7" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:19.370767 15755 raft_consensus.cc:3037] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.373196 15755 raft_consensus.cc:2454] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:19.375397 16907 consensus_queue.cc:1035] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.378571 16024 raft_consensus.cc:3037] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.380643 16092 leader_election.cc:302] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:19.381160 16745 raft_consensus.cc:2781] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.381925 16745 raft_consensus.cc:686] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:19.382298 16745 consensus_queue.cc:237] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.382805 16024 raft_consensus.cc:2454] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:19.385505 16909 tablet_bootstrap.cc:492] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:19.386268 16909 tablet_bootstrap.cc:492] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:12:19.386646 16909 ts_tablet_manager.cc:1367] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.245s user 0.060s sys 0.004s
I20240501 08:12:19.387840 16909 raft_consensus.cc:348] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.388199 16909 raft_consensus.cc:729] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.388592 16909 consensus_queue.cc:260] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.389370 16909 ts_tablet_manager.cc:1398] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:19.390952 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-cd9a5b5333d54f6d9d8186e4d9b1b1be received from {username='slave'} at 127.6.56.69:47087
I20240501 08:12:19.391093 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-cd9a5b5333d54f6d9d8186e4d9b1b1be on tablet cd9a5b5333d54f6d9d8186e4d9b1b1be with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:19.396891 12832 catalog_manager.cc:5484] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:19.426085 15755 raft_consensus.cc:1203] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:19.428606 15502 consensus_queue.cc:1035] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.440650 16927 tablet_bootstrap.cc:492] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:19.441092 16927 tablet_bootstrap.cc:492] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:12:19.441419 16927 ts_tablet_manager.cc:1367] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.239s user 0.065s sys 0.022s
I20240501 08:12:19.442284 16927 raft_consensus.cc:348] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.442577 16927 raft_consensus.cc:729] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.442883 16927 consensus_queue.cc:260] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.443495 16927 ts_tablet_manager.cc:1398] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:19.444720 13125 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-d5179549829d451eb3a6e2083ef59b7f received from {username='slave'} at 127.6.56.70:47641
I20240501 08:12:19.444878 13125 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 4d516916eeac468db514eeb928f06190-d5179549829d451eb3a6e2083ef59b7f on tablet d5179549829d451eb3a6e2083ef59b7f with peer 4d516916eeac468db514eeb928f06190
I20240501 08:12:19.501463 13364 consensus_queue.cc:237] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:19.503918 12974 raft_consensus.cc:1261] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:19.503978 13298 consensus_peers.cc:485] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9348009ae1444e73b6120161309dc4a2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:19.503995 13234 raft_consensus.cc:1261] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:19.504660 16977 consensus_queue.cc:1035] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:19.504988 16907 consensus_queue.cc:1035] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:19.509035 16024 raft_consensus.cc:1203] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:19.509438 12829 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:19.516569 15755 raft_consensus.cc:1261] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:19.517268 16473 consensus_queue.cc:1035] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.521153 12829 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 77e38e896cd5491b97cf470080d884b7 (table rebalancer_dynamic_table [id=969800c6b9e044e9b164ac3460f4e306]). Expected version 1 got 0
I20240501 08:12:19.522010 16024 raft_consensus.cc:1261] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:19.523052 16891 tablet.cc:1722] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:19.523561 16891 tablet.cc:1620] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:19.525208 12829 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 77e38e896cd5491b97cf470080d884b7 (table rebalancer_dynamic_table [id=969800c6b9e044e9b164ac3460f4e306]). Expected version 1 got 0
I20240501 08:12:19.530099 16745 consensus_queue.cc:1035] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.533059 16982 tablet.cc:1722] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:19.534454 16982 tablet.cc:1620] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:19.535562 16889 tablet.cc:1722] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:19.536039 16889 tablet.cc:1620] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:19.537676 12832 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet 77e38e896cd5491b97cf470080d884b7 (table rebalancer_dynamic_table [id=969800c6b9e044e9b164ac3460f4e306]). Expected version 1 got 0
I20240501 08:12:19.548180 12829 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=969800c6b9e044e9b164ac3460f4e306] alter complete (version 1)
I20240501 08:12:19.554963 16977 raft_consensus.cc:2932] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:19.555558 13235 raft_consensus.cc:2932] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:19.562093 12974 raft_consensus.cc:2932] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:19.563522 12829 catalog_manager.cc:5484] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:19.609828 15893 raft_consensus.cc:1203] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:19.612756 12974 consensus_queue.cc:237] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.618220 13234 raft_consensus.cc:1261] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:19.620074 16578 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:19.625766 13363 raft_consensus.cc:1261] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:19.626502 17004 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:19.629405 16578 raft_consensus.cc:2932] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
W20240501 08:12:19.631348 12909 consensus_peers.cc:485] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 427bc3b61a874f83bc27d9be24732e00. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:19.631744 13363 raft_consensus.cc:2932] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:19.634814 13234 raft_consensus.cc:2932] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:19.635787 12828 catalog_manager.cc:5484] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:19.654947 12828 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:19.662792 16982 tablet.cc:1722] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:19.663324 16982 tablet.cc:1620] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:19.665318 16889 tablet.cc:1722] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:19.665995 16889 tablet.cc:1620] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:19.668129 16891 tablet.cc:1722] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:19.668668 16891 tablet.cc:1620] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:19.676028 12828 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=969800c6b9e044e9b164ac3460f4e306] alter complete (version 2)
I20240501 08:12:19.689437 13363 consensus_queue.cc:237] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.691836 12974 raft_consensus.cc:1261] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:19.691819 13234 raft_consensus.cc:1261] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:19.692361 13298 consensus_peers.cc:485] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1af72684b304cab9a7d960084ea5926. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:19.692445 16985 consensus_queue.cc:1035] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:19.692770 16907 consensus_queue.cc:1035] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:19.696003 16985 raft_consensus.cc:2932] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:19.696640 12974 raft_consensus.cc:2932] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:19.699874 13234 raft_consensus.cc:2932] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:19.701956 12828 catalog_manager.cc:5484] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:19.706846 16495 consensus_queue.cc:1035] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.713430 16936 consensus_queue.cc:1035] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:19.731938 12832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:19.742498 12954 tablet_service.cc:1467] Processing CreateTablet for tablet fc4ad07cecff43f0b867e4db444957e7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.743000 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4ad07cecff43f0b867e4db444957e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.743058 15872 tablet_service.cc:1467] Processing CreateTablet for tablet fc4ad07cecff43f0b867e4db444957e7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.743469 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4ad07cecff43f0b867e4db444957e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.749044 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 15dd4768e5ff47719db40ad39fbd2838 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.749029 16005 tablet_service.cc:1467] Processing CreateTablet for tablet fc4ad07cecff43f0b867e4db444957e7 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.749454 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15dd4768e5ff47719db40ad39fbd2838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.749468 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4ad07cecff43f0b867e4db444957e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.750877 17044 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:19.752575 17044 tablet_bootstrap.cc:654] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.753746 13363 consensus_queue.cc:237] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:19.756438 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 15dd4768e5ff47719db40ad39fbd2838 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.756923 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15dd4768e5ff47719db40ad39fbd2838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.758174 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 82e08735b5db41dcbccec46aa7278530 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.758534 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e08735b5db41dcbccec46aa7278530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.761512 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 82e08735b5db41dcbccec46aa7278530 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.762099 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e08735b5db41dcbccec46aa7278530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.762770 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 15dd4768e5ff47719db40ad39fbd2838 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.763234 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15dd4768e5ff47719db40ad39fbd2838. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.769834 16945 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:19.771436 16004 tablet_service.cc:1467] Processing CreateTablet for tablet 82e08735b5db41dcbccec46aa7278530 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:19.771903 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e08735b5db41dcbccec46aa7278530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:19.771955 16945 tablet_bootstrap.cc:654] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.777427 13234 raft_consensus.cc:1261] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:19.778097 16985 consensus_queue.cc:1035] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:19.778863 16888 tablet_bootstrap.cc:492] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:19.780848 16888 tablet_bootstrap.cc:654] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.780905 12832 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:19.789970 16982 tablet.cc:1722] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:19.801389 16888 tablet_bootstrap.cc:492] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:19.801643 16888 ts_tablet_manager.cc:1367] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.002s
I20240501 08:12:19.801717 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=969800c6b9e044e9b164ac3460f4e306] alter complete (version 3)
I20240501 08:12:19.802824 16888 raft_consensus.cc:348] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.804958 17052 tablet_bootstrap.cc:492] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:19.807305 13104 raft_consensus.cc:1261] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:19.808336 16985 consensus_queue.cc:1035] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:19.808557 16889 tablet.cc:1722] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:19.808115 16891 tablet.cc:1722] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:19.812263 17052 tablet_bootstrap.cc:654] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.813792 16888 raft_consensus.cc:374] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.813997 16888 raft_consensus.cc:729] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.814375 16888 consensus_queue.cc:260] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.815191 16888 ts_tablet_manager.cc:1398] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240501 08:12:19.821997 17052 tablet_bootstrap.cc:492] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:19.824620 17052 ts_tablet_manager.cc:1367] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 08:12:19.825668 17044 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:19.826610 16945 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:19.826843 16945 ts_tablet_manager.cc:1367] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.013s
I20240501 08:12:19.827864 16945 raft_consensus.cc:348] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:19.828178 16945 raft_consensus.cc:374] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.828272 16945 raft_consensus.cc:729] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.828588 16945 consensus_queue.cc:260] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:19.829205 16945 ts_tablet_manager.cc:1398] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:19.829319 17047 tablet_bootstrap.cc:492] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
W20240501 08:12:19.829339 13300 consensus_peers.cc:485] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa5b3517ffc490cab68d09bbe2d5c0e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:19.830037 16945 tablet_bootstrap.cc:492] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:19.832690 16945 tablet_bootstrap.cc:654] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.832211 16907 raft_consensus.cc:2932] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:19.833098 17044 ts_tablet_manager.cc:1367] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.082s user 0.017s sys 0.009s
I20240501 08:12:19.833863 17047 tablet_bootstrap.cc:654] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.834787 17044 raft_consensus.cc:348] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:19.835028 17044 raft_consensus.cc:374] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.835108 17044 raft_consensus.cc:729] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.835664 17044 consensus_queue.cc:260] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:19.839581 17044 ts_tablet_manager.cc:1398] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:19.837929 13234 raft_consensus.cc:2932] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:19.842053 17047 tablet_bootstrap.cc:492] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:19.842209 17047 ts_tablet_manager.cc:1367] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240501 08:12:19.842911 17047 raft_consensus.cc:348] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.843089 17047 raft_consensus.cc:374] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.843184 17047 raft_consensus.cc:729] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.843415 17047 consensus_queue.cc:260] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.843863 17047 ts_tablet_manager.cc:1398] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:19.844158 17047 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:19.845852 17047 tablet_bootstrap.cc:654] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.846686 16945 tablet_bootstrap.cc:492] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:19.847020 16945 ts_tablet_manager.cc:1367] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20240501 08:12:19.850245 16945 raft_consensus.cc:348] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.850672 16945 raft_consensus.cc:374] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.850879 16945 raft_consensus.cc:729] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.851284 16945 consensus_queue.cc:260] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.851905 12829 catalog_manager.cc:5484] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:19.840880 17044 tablet_bootstrap.cc:492] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:19.853929 16945 ts_tablet_manager.cc:1398] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:12:19.854074 17047 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:19.854271 17047 ts_tablet_manager.cc:1367] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240501 08:12:19.855770 17047 raft_consensus.cc:348] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:19.856045 17047 raft_consensus.cc:374] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.856163 17047 raft_consensus.cc:729] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.856506 17047 consensus_queue.cc:260] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:19.856989 17047 ts_tablet_manager.cc:1398] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:12:19.860980 13104 raft_consensus.cc:2932] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:19.856617 17052 raft_consensus.cc:348] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.866262 17052 raft_consensus.cc:374] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.866401 17052 raft_consensus.cc:729] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.866636 17052 consensus_queue.cc:260] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.867112 17052 ts_tablet_manager.cc:1398] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20240501 08:12:19.867437 17052 tablet_bootstrap.cc:492] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:19.868970 17044 tablet_bootstrap.cc:654] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.874791 17052 tablet_bootstrap.cc:654] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:19.886161 17052 tablet_bootstrap.cc:492] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:19.886387 17052 ts_tablet_manager.cc:1367] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20240501 08:12:19.887408 17052 raft_consensus.cc:348] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.887681 17052 raft_consensus.cc:374] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.887737 17052 raft_consensus.cc:729] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.892592 17052 consensus_queue.cc:260] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.893412 17052 ts_tablet_manager.cc:1398] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 08:12:19.909119 12832 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:19.909385 12832 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:19.913000 17044 tablet_bootstrap.cc:492] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:19.913244 17044 ts_tablet_manager.cc:1367] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20240501 08:12:19.914254 17044 raft_consensus.cc:348] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.914472 17044 raft_consensus.cc:374] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:19.914537 17044 raft_consensus.cc:729] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:19.914772 17044 consensus_queue.cc:260] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.915321 17044 ts_tablet_manager.cc:1398] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:19.915475 17048 raft_consensus.cc:482] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.915637 17048 raft_consensus.cc:504] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:19.916481 17048 leader_election.cc:288] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:19.918308 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4ad07cecff43f0b867e4db444957e7" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:19.918605 15893 raft_consensus.cc:2454] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:19.919330 15957 leader_election.cc:302] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:19.919368 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 77e38e896cd5491b97cf470080d884b7
I20240501 08:12:19.919745 17048 raft_consensus.cc:2781] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.919898 17048 raft_consensus.cc:482] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.920015 17048 raft_consensus.cc:3037] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.920925 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e38e896cd5491b97cf470080d884b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:19 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:19.920933 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e38e896cd5491b97cf470080d884b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:19 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:19.922633 16004 tablet_service.cc:1514] Processing DeleteTablet for tablet 77e38e896cd5491b97cf470080d884b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:19 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:19.925416 17048 raft_consensus.cc:504] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:19.926719 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4ad07cecff43f0b867e4db444957e7" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:19.926992 15893 raft_consensus.cc:3037] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.930176 15893 raft_consensus.cc:2454] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:19.932438 15957 leader_election.cc:302] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:19.932844 16826 raft_consensus.cc:2781] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.935788 17048 leader_election.cc:288] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:19.938673 16952 raft_consensus.cc:482] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.938838 16952 raft_consensus.cc:504] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:19.939837 16952 leader_election.cc:288] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:19.954083 16793 raft_consensus.cc:482] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.954309 16793 raft_consensus.cc:504] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.955418 16793 leader_election.cc:288] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:19.969957 17077 raft_consensus.cc:482] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:19.970263 17077 raft_consensus.cc:504] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.973169 17077 leader_election.cc:288] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:19.973380 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e08735b5db41dcbccec46aa7278530" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:19.973644 15755 raft_consensus.cc:2454] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:19.973634 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e08735b5db41dcbccec46aa7278530" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:19.973857 13234 raft_consensus.cc:2454] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:19.974303 15959 leader_election.cc:302] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:19.977074 16826 raft_consensus.cc:686] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:19.977558 16826 consensus_queue.cc:237] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:19.978986 17077 raft_consensus.cc:2781] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:19.979175 17077 raft_consensus.cc:482] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:19.979341 17077 raft_consensus.cc:3037] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.980962 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e08735b5db41dcbccec46aa7278530" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:19.982478 17077 raft_consensus.cc:504] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.987003 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4ad07cecff43f0b867e4db444957e7" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:19.987257 12972 raft_consensus.cc:2454] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:19.987612 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc4ad07cecff43f0b867e4db444957e7" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:19.987774 12974 raft_consensus.cc:3037] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.989440 16025 raft_consensus.cc:2379] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ed448d04e4646658c0de37d829ff15b in current term 1: Already voted for candidate 5dd722c770d34d6c9154a87917117161 in this term.
I20240501 08:12:19.991246 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e08735b5db41dcbccec46aa7278530" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:19.991516 17077 leader_election.cc:288] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:19.991549 15755 raft_consensus.cc:3037] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.993862 12974 raft_consensus.cc:2454] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:19.994729 15755 raft_consensus.cc:2454] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:19.995000 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e08735b5db41dcbccec46aa7278530" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:19.995165 15959 leader_election.cc:302] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:19.995199 13235 raft_consensus.cc:3037] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:19.997308 17077 raft_consensus.cc:2781] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:19.997727 17077 raft_consensus.cc:686] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:19.998288 17077 consensus_queue.cc:237] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:19.999902 13235 raft_consensus.cc:2454] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:20.002766 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15dd4768e5ff47719db40ad39fbd2838" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:20.003007 13235 raft_consensus.cc:2454] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:20.003458 15826 leader_election.cc:302] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:20.003844 16952 raft_consensus.cc:2781] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.003958 16952 raft_consensus.cc:482] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.004026 16952 raft_consensus.cc:3037] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.005249 17075 tablet_replica.cc:330] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:20.005450 17075 raft_consensus.cc:2229] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:20.005795 17075 raft_consensus.cc:2258] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.006769 16952 raft_consensus.cc:504] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:20.007449 16952 leader_election.cc:288] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:20.008608 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15dd4768e5ff47719db40ad39fbd2838" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:20.008832 13235 raft_consensus.cc:3037] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.013411 13235 raft_consensus.cc:2454] T 15dd4768e5ff47719db40ad39fbd2838 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:20.013882 15826 leader_election.cc:302] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:20.014250 16952 raft_consensus.cc:2781] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.015219 17075 ts_tablet_manager.cc:1875] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:20.016240 17074 tablet_replica.cc:330] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:20.016597 17074 raft_consensus.cc:2229] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:20.016914 17074 raft_consensus.cc:2258] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.017961 17074 ts_tablet_manager.cc:1875] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:20.023336 12832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:20.030112 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15dd4768e5ff47719db40ad39fbd2838" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:20.030404 12974 raft_consensus.cc:2454] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:20.031487 17075 ts_tablet_manager.cc:1888] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:20.031701 17075 log.cc:1196] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/77e38e896cd5491b97cf470080d884b7
I20240501 08:12:20.032004 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15dd4768e5ff47719db40ad39fbd2838" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:20.032240 12974 raft_consensus.cc:3037] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.032541 17075 ts_tablet_manager.cc:1909] T 77e38e896cd5491b97cf470080d884b7 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:20.034771 12974 raft_consensus.cc:2454] T 15dd4768e5ff47719db40ad39fbd2838 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:20.035229 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 77e38e896cd5491b97cf470080d884b7 (table rebalancer_dynamic_table_new_name [id=969800c6b9e044e9b164ac3460f4e306]) successfully deleted
I20240501 08:12:20.036891 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5382759c9bec4cb99561518cbc7c7330]), partition=
I20240501 08:12:20.037047 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5382759c9bec4cb99561518cbc7c7330]), partition=
I20240501 08:12:20.037313 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.037429 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.043265 12829 catalog_manager.cc:5484] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:20.048269 17076 tablet_replica.cc:330] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:20.048547 17076 raft_consensus.cc:2229] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:20.048816 17076 raft_consensus.cc:2258] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.048805 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e08735b5db41dcbccec46aa7278530" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:20.049137 15755 raft_consensus.cc:2379] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ed448d04e4646658c0de37d829ff15b in current term 1: Already voted for candidate 5dd722c770d34d6c9154a87917117161 in this term.
I20240501 08:12:20.049547 13170 leader_election.cc:302] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b; no voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:20.050035 17076 ts_tablet_manager.cc:1875] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:20.050045 16224 raft_consensus.cc:2726] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:20.052647 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5382759c9bec4cb99561518cbc7c7330]), partition=
I20240501 08:12:20.053236 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.057742 12829 catalog_manager.cc:5484] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:20.065203 17044 tablet_bootstrap.cc:492] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:20.066795 17044 tablet_bootstrap.cc:654] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.067637 17076 ts_tablet_manager.cc:1888] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:20.067823 16952 raft_consensus.cc:686] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:20.067853 17074 ts_tablet_manager.cc:1888] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:20.067994 17074 log.cc:1196] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/77e38e896cd5491b97cf470080d884b7
I20240501 08:12:20.068197 16952 consensus_queue.cc:237] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:20.069145 17074 ts_tablet_manager.cc:1909] T 77e38e896cd5491b97cf470080d884b7 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:20.073262 17076 log.cc:1196] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/77e38e896cd5491b97cf470080d884b7
I20240501 08:12:20.071153 16888 tablet_bootstrap.cc:492] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:20.074030 17076 ts_tablet_manager.cc:1909] T 77e38e896cd5491b97cf470080d884b7 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:20.075507 16888 tablet_bootstrap.cc:654] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.079238 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 77e38e896cd5491b97cf470080d884b7 (table rebalancer_dynamic_table_new_name [id=969800c6b9e044e9b164ac3460f4e306]) successfully deleted
I20240501 08:12:20.080847 17086 tablet_bootstrap.cc:492] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:20.081009 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 77e38e896cd5491b97cf470080d884b7 (table rebalancer_dynamic_table_new_name [id=969800c6b9e044e9b164ac3460f4e306]) successfully deleted
I20240501 08:12:20.082818 17086 tablet_bootstrap.cc:654] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.084007 17091 ts_tablet_manager.cc:902] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:20.089277 16888 tablet_bootstrap.cc:492] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:20.089582 16888 ts_tablet_manager.cc:1367] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20240501 08:12:20.091066 16888 raft_consensus.cc:348] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.091462 16888 raft_consensus.cc:374] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.091606 16888 raft_consensus.cc:729] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.092000 16888 consensus_queue.cc:260] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.096410 16888 ts_tablet_manager.cc:1398] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240501 08:12:20.096534 17086 tablet_bootstrap.cc:492] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:20.096830 17086 ts_tablet_manager.cc:1367] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20240501 08:12:20.097815 17086 raft_consensus.cc:348] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.098114 17086 raft_consensus.cc:374] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.098246 17086 raft_consensus.cc:729] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.098539 17086 consensus_queue.cc:260] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.099166 17086 ts_tablet_manager.cc:1398] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:20.113952 17044 tablet_bootstrap.cc:492] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:20.114298 17044 ts_tablet_manager.cc:1367] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.049s user 0.001s sys 0.011s
I20240501 08:12:20.115162 17044 raft_consensus.cc:348] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.116889 17044 raft_consensus.cc:374] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.117730 17044 raft_consensus.cc:729] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.118023 17044 consensus_queue.cc:260] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.118124 12825 catalog_manager.cc:5484] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:20.118729 17044 ts_tablet_manager.cc:1398] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:20.124047 17091 tablet_copy_client.cc:308] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:12:20.124826 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 9348009ae1444e73b6120161309dc4a2 from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:50899)
I20240501 08:12:20.125000 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 9348009ae1444e73b6120161309dc4a2 from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:50899: session id = 6cc32f6e2d49477f9a23ce6e444039fb-9348009ae1444e73b6120161309dc4a2
I20240501 08:12:20.127522 13385 tablet_copy_source_session.cc:205] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:20.129596 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9348009ae1444e73b6120161309dc4a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.134872 17091 tablet_copy_client.cc:783] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:20.139081 17091 tablet_copy_client.cc:647] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:20.141141 17091 tablet_copy_client.cc:523] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:20.144937 17091 tablet_bootstrap.cc:492] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:20.148671 13235 consensus_queue.cc:237] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:20.151885 12974 raft_consensus.cc:1261] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:20.152662 13364 raft_consensus.cc:1261] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:20.153489 16793 consensus_queue.cc:1035] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:20.154327 16224 consensus_queue.cc:1035] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:20.201069 17108 ts_tablet_manager.cc:902] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:20.213088 17107 ts_tablet_manager.cc:902] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:20.214605 16793 raft_consensus.cc:2932] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:20.220248 17091 tablet_bootstrap.cc:492] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:20.219914 12974 raft_consensus.cc:2932] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:20.220180 13364 raft_consensus.cc:2932] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:20.220861 17091 tablet_bootstrap.cc:492] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:12:20.221200 17091 ts_tablet_manager.cc:1367] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.076s user 0.060s sys 0.005s
I20240501 08:12:20.222301 17091 raft_consensus.cc:348] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:20.222617 17091 raft_consensus.cc:729] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.223737 17091 consensus_queue.cc:260] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:20.224370 12829 catalog_manager.cc:5484] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:20.226611 17091 ts_tablet_manager.cc:1398] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240501 08:12:20.227229 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-9348009ae1444e73b6120161309dc4a2 received from {username='slave'} at 127.6.56.72:50899
I20240501 08:12:20.227373 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-9348009ae1444e73b6120161309dc4a2 on tablet 9348009ae1444e73b6120161309dc4a2 with peer 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:20.238857 13168 consensus_peers.cc:485] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7883772f69cd4a18a2a6d568adbdeb61. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.240587 17108 tablet_copy_client.cc:308] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:12:20.241585 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet d1af72684b304cab9a7d960084ea5926 from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:45907)
I20240501 08:12:20.241756 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet d1af72684b304cab9a7d960084ea5926 from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:45907: session id = 4d516916eeac468db514eeb928f06190-d1af72684b304cab9a7d960084ea5926
I20240501 08:12:20.244055 13385 tablet_copy_source_session.cc:205] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:20.246380 17108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1af72684b304cab9a7d960084ea5926. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.249382 16495 raft_consensus.cc:482] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.249565 16495 raft_consensus.cc:504] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.250237 16495 leader_election.cc:288] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:20.250716 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efda0ff8d8c4e71acd4ea20c2e772f5" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:20.250950 15893 raft_consensus.cc:2454] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:20.251013 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efda0ff8d8c4e71acd4ea20c2e772f5" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:20.251241 13104 raft_consensus.cc:2454] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:20.251483 17108 tablet_copy_client.cc:783] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:20.251703 17108 tablet_copy_client.cc:647] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:20.251648 15689 leader_election.cc:302] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:20.252503 16495 raft_consensus.cc:2781] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.252606 16495 raft_consensus.cc:482] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.252717 16495 raft_consensus.cc:3037] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.255661 17108 tablet_copy_client.cc:523] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:20.256280 16495 raft_consensus.cc:504] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.257414 16495 leader_election.cc:288] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:20.257593 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efda0ff8d8c4e71acd4ea20c2e772f5" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:20.257694 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7efda0ff8d8c4e71acd4ea20c2e772f5" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:20.257809 15893 raft_consensus.cc:3037] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.257957 13104 raft_consensus.cc:3037] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.259014 17108 tablet_bootstrap.cc:492] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:20.260746 13104 raft_consensus.cc:2454] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:20.260795 15893 raft_consensus.cc:2454] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:20.261720 15689 leader_election.cc:302] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:20.262099 16495 raft_consensus.cc:2781] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.263383 17107 tablet_copy_client.cc:308] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:12:20.264357 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet 427bc3b61a874f83bc27d9be24732e00 from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:33711)
I20240501 08:12:20.264542 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet 427bc3b61a874f83bc27d9be24732e00 from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:33711: session id = 5dd722c770d34d6c9154a87917117161-427bc3b61a874f83bc27d9be24732e00
I20240501 08:12:20.267607 12994 tablet_copy_source_session.cc:205] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:20.269872 17107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427bc3b61a874f83bc27d9be24732e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.275030 16495 raft_consensus.cc:686] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:20.275372 16495 consensus_queue.cc:237] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.280390 17107 tablet_copy_client.cc:783] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:20.280853 17107 tablet_copy_client.cc:647] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:20.284359 17107 tablet_copy_client.cc:523] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:20.286587 12825 catalog_manager.cc:5484] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:20.290468 17107 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:20.336383 17108 tablet_bootstrap.cc:492] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:20.338068 17108 tablet_bootstrap.cc:492] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:12:20.338387 17108 ts_tablet_manager.cc:1367] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.079s user 0.050s sys 0.005s
I20240501 08:12:20.339155 17108 raft_consensus.cc:348] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:20.339411 17108 raft_consensus.cc:729] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.339686 17108 consensus_queue.cc:260] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:20.340524 17108 ts_tablet_manager.cc:1398] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:20.341459 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-d1af72684b304cab9a7d960084ea5926 received from {username='slave'} at 127.6.56.70:45907
I20240501 08:12:20.341622 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 4d516916eeac468db514eeb928f06190-d1af72684b304cab9a7d960084ea5926 on tablet d1af72684b304cab9a7d960084ea5926 with peer 4d516916eeac468db514eeb928f06190
I20240501 08:12:20.348702 17131 ts_tablet_manager.cc:902] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:20.356439 17107 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:20.357185 17107 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:12:20.357576 17107 ts_tablet_manager.cc:1367] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.067s user 0.046s sys 0.011s
I20240501 08:12:20.358557 17107 raft_consensus.cc:348] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:20.358894 17107 raft_consensus.cc:729] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.359339 17107 consensus_queue.cc:260] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:20.360356 17107 ts_tablet_manager.cc:1398] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:20.361340 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-427bc3b61a874f83bc27d9be24732e00 received from {username='slave'} at 127.6.56.71:33711
I20240501 08:12:20.361503 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 5dd722c770d34d6c9154a87917117161-427bc3b61a874f83bc27d9be24732e00 on tablet 427bc3b61a874f83bc27d9be24732e00 with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:12:20.362269 17131 tablet_copy_client.cc:308] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:12:20.368678 12974 consensus_queue.cc:237] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:20.373996 13364 raft_consensus.cc:1261] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:20.374307 13235 raft_consensus.cc:1261] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:20.374487 12909 consensus_peers.cc:485] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c4b83a76d7b44039ebb17245fcd8a3c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.374939 16578 consensus_queue.cc:1035] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:20.375290 17004 consensus_queue.cc:1035] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:20.378700 17004 raft_consensus.cc:2932] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:20.379727 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet faa5b3517ffc490cab68d09bbe2d5c0e from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:46029)
I20240501 08:12:20.379851 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet faa5b3517ffc490cab68d09bbe2d5c0e from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:46029: session id = 982f49fc87f149439cda0de2261b6bfd-faa5b3517ffc490cab68d09bbe2d5c0e
I20240501 08:12:20.381693 13235 raft_consensus.cc:2932] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:20.382181 13385 tablet_copy_source_session.cc:205] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:20.384472 16826 consensus_queue.cc:1035] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.387806 17131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faa5b3517ffc490cab68d09bbe2d5c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.392239 12825 catalog_manager.cc:5484] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:20.395809 17131 tablet_copy_client.cc:783] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:20.396173 17131 tablet_copy_client.cc:647] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:20.398347 17131 tablet_copy_client.cc:523] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:20.402575 17131 tablet_bootstrap.cc:492] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:20.407706 16826 consensus_queue.cc:1035] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.411496 16826 consensus_queue.cc:1035] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.427446 12825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:20.428141 16826 consensus_queue.cc:1035] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.451725 13363 raft_consensus.cc:2932] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:20.480479 15893 raft_consensus.cc:1261] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:20.481226 16495 consensus_queue.cc:1035] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.482666 13104 raft_consensus.cc:1261] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:20.483310 16495 consensus_queue.cc:1035] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.495415 12832 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (table rebalancer_dynamic_table [id=5382759c9bec4cb99561518cbc7c7330]). Expected version 1 got 0
I20240501 08:12:20.515519 17169 tablet.cc:1722] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:20.518381 12832 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (table rebalancer_dynamic_table [id=5382759c9bec4cb99561518cbc7c7330]). Expected version 1 got 0
I20240501 08:12:20.526616 17168 tablet.cc:1722] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:20.527926 17168 tablet.cc:1620] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:20.536249 17169 tablet.cc:1620] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:20.543035 12829 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (table rebalancer_dynamic_table [id=5382759c9bec4cb99561518cbc7c7330]). Expected version 1 got 0
I20240501 08:12:20.543489 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=5382759c9bec4cb99561518cbc7c7330] alter complete (version 1)
I20240501 08:12:20.550844 17174 tablet.cc:1722] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:20.551472 17174 tablet.cc:1620] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:20.554415 12829 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (table rebalancer_dynamic_table [id=5382759c9bec4cb99561518cbc7c7330]). Expected version 1 got 0
I20240501 08:12:20.560492 17131 tablet_bootstrap.cc:492] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:20.561026 17131 tablet_bootstrap.cc:492] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:12:20.561385 17131 ts_tablet_manager.cc:1367] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.159s user 0.060s sys 0.005s
I20240501 08:12:20.562475 17131 raft_consensus.cc:348] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.562799 17131 raft_consensus.cc:729] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.563094 17131 consensus_queue.cc:260] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:20.563853 17131 ts_tablet_manager.cc:1398] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:20.564662 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-faa5b3517ffc490cab68d09bbe2d5c0e received from {username='slave'} at 127.6.56.69:46029
I20240501 08:12:20.564805 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-faa5b3517ffc490cab68d09bbe2d5c0e on tablet faa5b3517ffc490cab68d09bbe2d5c0e with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:20.574517 16952 consensus_queue.cc:1035] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.577978 16952 consensus_queue.cc:1035] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:20.624329 16025 consensus_queue.cc:237] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:20.626814 13104 raft_consensus.cc:1261] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:20.626899 15755 raft_consensus.cc:1261] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:20.627283 15960 consensus_peers.cc:485] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a48ba4666c14848b2b5a1e62f16357b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.627383 16826 consensus_queue.cc:1035] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:20.627687 17048 consensus_queue.cc:1035] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:20.630275 16826 raft_consensus.cc:2932] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:20.630877 15755 raft_consensus.cc:2932] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:20.636008 12829 catalog_manager.cc:5484] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:20.634529 13104 raft_consensus.cc:2932] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:20.642215 17091 ts_tablet_manager.cc:902] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:20.652633 12832 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:20.656111 12825 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:20.656653 17091 tablet_copy_client.cc:308] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:12:20.657375 13255 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet 7883772f69cd4a18a2a6d568adbdeb61 from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:54825)
I20240501 08:12:20.657549 13255 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet 7883772f69cd4a18a2a6d568adbdeb61 from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:54825: session id = 6cc32f6e2d49477f9a23ce6e444039fb-7883772f69cd4a18a2a6d568adbdeb61
I20240501 08:12:20.660259 13255 tablet_copy_source_session.cc:205] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:20.662999 17091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7883772f69cd4a18a2a6d568adbdeb61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.664070 17168 tablet.cc:1722] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:20.664752 17168 tablet.cc:1620] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:20.666314 17169 tablet.cc:1722] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:20.666817 17169 tablet.cc:1620] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:20.669175 17091 tablet_copy_client.cc:783] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:20.669404 17091 tablet_copy_client.cc:647] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:20.672811 17091 tablet_copy_client.cc:523] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:20.676144 17091 tablet_bootstrap.cc:492] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:20.679141 17174 tablet.cc:1722] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:20.679634 17174 tablet.cc:1620] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:20.679777 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=5382759c9bec4cb99561518cbc7c7330] alter complete (version 2)
I20240501 08:12:20.683048 13345 tablet_service.cc:1467] Processing CreateTablet for tablet f22fc6797cb24451b78a5c1e5f6306ba (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.683521 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22fc6797cb24451b78a5c1e5f6306ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.683804 15735 tablet_service.cc:1467] Processing CreateTablet for tablet cf908880298b462391a84f5a8d4e3a6b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.684149 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf908880298b462391a84f5a8d4e3a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.684711 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 55b77e87381d402ab836985100ccb279 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.685137 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b77e87381d402ab836985100ccb279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.688736 16138 tablet_service.cc:1467] Processing CreateTablet for tablet f22fc6797cb24451b78a5c1e5f6306ba (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.688762 12954 tablet_service.cc:1467] Processing CreateTablet for tablet cf908880298b462391a84f5a8d4e3a6b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.689172 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf908880298b462391a84f5a8d4e3a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.689193 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22fc6797cb24451b78a5c1e5f6306ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.689642 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 55b77e87381d402ab836985100ccb279 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.690048 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b77e87381d402ab836985100ccb279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.695493 15893 raft_consensus.cc:1203] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:20.695631 16137 tablet_service.cc:1467] Processing CreateTablet for tablet cf908880298b462391a84f5a8d4e3a6b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.696069 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf908880298b462391a84f5a8d4e3a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.697317 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 55b77e87381d402ab836985100ccb279 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.698671 16157 raft_consensus.cc:1203] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:20.702838 12953 tablet_service.cc:1467] Processing CreateTablet for tablet f22fc6797cb24451b78a5c1e5f6306ba (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:20.703431 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22fc6797cb24451b78a5c1e5f6306ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.708827 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55b77e87381d402ab836985100ccb279. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.725441 17209 tablet_bootstrap.cc:492] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:20.728332 17209 tablet_bootstrap.cc:654] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.751025 16025 raft_consensus.cc:1203] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:20.753242 17213 tablet_bootstrap.cc:492] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:20.753538 17214 tablet_bootstrap.cc:492] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:20.756767 17214 tablet_bootstrap.cc:654] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.756909 17213 tablet_bootstrap.cc:654] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.760617 17212 tablet_bootstrap.cc:492] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:20.762882 17212 tablet_bootstrap.cc:654] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.767665 17214 tablet_bootstrap.cc:492] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:20.768595 17214 ts_tablet_manager.cc:1367] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20240501 08:12:20.769347 17211 tablet_bootstrap.cc:492] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:20.769801 17107 ts_tablet_manager.cc:902] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:20.769757 17214 raft_consensus.cc:348] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:20.769989 17214 raft_consensus.cc:374] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.770088 17214 raft_consensus.cc:729] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.770316 17214 consensus_queue.cc:260] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
W20240501 08:12:20.770572 13301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 73558us
I20240501 08:12:20.772807 17214 ts_tablet_manager.cc:1398] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:20.773705 17214 tablet_bootstrap.cc:492] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:20.775835 17214 tablet_bootstrap.cc:654] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.784924 17211 tablet_bootstrap.cc:654] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.785494 17214 tablet_bootstrap.cc:492] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:20.786015 17214 ts_tablet_manager.cc:1367] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240501 08:12:20.787158 17214 raft_consensus.cc:348] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.787550 17205 tablet_bootstrap.cc:492] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:20.788178 17214 raft_consensus.cc:374] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.788357 17214 raft_consensus.cc:729] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.788681 17214 consensus_queue.cc:260] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.789678 17214 ts_tablet_manager.cc:1398] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:20.790974 17205 tablet_bootstrap.cc:654] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.792372 12825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:20.795256 17147 raft_consensus.cc:482] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.795408 17147 raft_consensus.cc:504] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.796276 17147 leader_election.cc:288] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:20.797103 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
W20240501 08:12:20.797958 12909 leader_election.cc:341] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:20.805702 17212 tablet_bootstrap.cc:492] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:20.806015 17212 ts_tablet_manager.cc:1367] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240501 08:12:20.806978 17212 raft_consensus.cc:348] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:20.809397 17212 raft_consensus.cc:374] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.809430 17168 tablet.cc:1722] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:20.810041 17212 raft_consensus.cc:729] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.811142 17212 consensus_queue.cc:260] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:20.813802 17212 ts_tablet_manager.cc:1398] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:12:20.816377 17174 tablet.cc:1722] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:20.817478 17169 tablet.cc:1722] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:20.819233 17107 tablet_copy_client.cc:308] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:12:20.820480 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet 7c4b83a76d7b44039ebb17245fcd8a3c from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:33711)
I20240501 08:12:20.820659 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet 7c4b83a76d7b44039ebb17245fcd8a3c from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:33711: session id = 5dd722c770d34d6c9154a87917117161-7c4b83a76d7b44039ebb17245fcd8a3c
I20240501 08:12:20.823722 12994 tablet_copy_source_session.cc:205] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:20.828433 17107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4b83a76d7b44039ebb17245fcd8a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:20.831595 12829 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=5382759c9bec4cb99561518cbc7c7330] alter complete (version 3)
I20240501 08:12:20.836500 17213 tablet_bootstrap.cc:492] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:20.837538 17205 tablet_bootstrap.cc:492] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:20.839249 17205 ts_tablet_manager.cc:1367] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20240501 08:12:20.839777 17213 ts_tablet_manager.cc:1367] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.087s user 0.005s sys 0.000s
I20240501 08:12:20.841017 17213 raft_consensus.cc:348] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:20.842742 17213 raft_consensus.cc:374] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.843075 17213 raft_consensus.cc:729] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.843886 17213 consensus_queue.cc:260] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:20.844780 17107 tablet_copy_client.cc:783] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:20.844671 17205 raft_consensus.cc:348] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:20.845042 17205 raft_consensus.cc:374] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.845067 17107 tablet_copy_client.cc:647] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:20.845237 17205 raft_consensus.cc:729] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.846572 17211 tablet_bootstrap.cc:492] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:20.846834 17211 ts_tablet_manager.cc:1367] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.004s
I20240501 08:12:20.847097 17107 tablet_copy_client.cc:523] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:20.848078 17213 ts_tablet_manager.cc:1398] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 08:12:20.850248 17211 raft_consensus.cc:348] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:20.850587 17209 tablet_bootstrap.cc:492] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:20.850705 17211 raft_consensus.cc:374] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.850755 17209 ts_tablet_manager.cc:1367] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.125s user 0.011s sys 0.035s
I20240501 08:12:20.850904 17211 raft_consensus.cc:729] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.851372 17211 consensus_queue.cc:260] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:20.851656 17209 raft_consensus.cc:348] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:20.851936 17209 raft_consensus.cc:374] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.852046 17209 raft_consensus.cc:729] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.852406 17209 consensus_queue.cc:260] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:20.854054 17205 consensus_queue.cc:260] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:20.854396 17209 ts_tablet_manager.cc:1398] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:20.854657 17209 tablet_bootstrap.cc:492] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:20.856374 17209 tablet_bootstrap.cc:654] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.856364 17211 ts_tablet_manager.cc:1398] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240501 08:12:20.857399 17211 tablet_bootstrap.cc:492] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:20.857065 17205 ts_tablet_manager.cc:1398] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 08:12:20.863983 17211 tablet_bootstrap.cc:654] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:20.880005 17107 tablet_bootstrap.cc:492] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:20.895781 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
W20240501 08:12:20.896651 12907 leader_election.cc:341] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:20.896813 12907 leader_election.cc:302] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2; no voters: 6cc32f6e2d49477f9a23ce6e444039fb, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:20.897249 17147 raft_consensus.cc:2726] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:20.900977 17211 tablet_bootstrap.cc:492] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:20.901878 17209 tablet_bootstrap.cc:492] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:20.902487 17209 ts_tablet_manager.cc:1367] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.002s
I20240501 08:12:20.903394 17209 raft_consensus.cc:348] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.903740 17209 raft_consensus.cc:374] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.903831 17209 raft_consensus.cc:729] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.904109 17209 consensus_queue.cc:260] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.904753 17211 ts_tablet_manager.cc:1367] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.007s
I20240501 08:12:20.905473 17209 ts_tablet_manager.cc:1398] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:20.907639 17211 raft_consensus.cc:348] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.910148 17211 raft_consensus.cc:374] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:20.910302 17211 raft_consensus.cc:729] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:20.910645 17211 consensus_queue.cc:260] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.910795 16025 consensus_queue.cc:237] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.914119 17211 ts_tablet_manager.cc:1398] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:20.919643 15755 raft_consensus.cc:1203] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 08:12:20.922817 15960 consensus_peers.cc:485] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bdabd94a46dc4a5796949fdf9d0c698c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:20.923364 15755 raft_consensus.cc:1261] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:20.924018 17156 consensus_queue.cc:1035] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:20.924197 15893 raft_consensus.cc:1261] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:20.926272 17077 consensus_queue.cc:1035] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:20.928223 17120 raft_consensus.cc:482] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:20.928400 17120 raft_consensus.cc:504] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.929077 17120 leader_election.cc:288] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:20.929730 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:20.930074 12972 raft_consensus.cc:2454] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:20.930529 15692 leader_election.cc:302] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:20.930788 17120 raft_consensus.cc:2781] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:20.930896 17120 raft_consensus.cc:482] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:20.930981 17120 raft_consensus.cc:3037] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.930881 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:20.931169 16157 raft_consensus.cc:2454] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:20.938128 17120 raft_consensus.cc:504] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.939235 17120 leader_election.cc:288] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:20.939462 12972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:20.939764 12972 raft_consensus.cc:3037] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.939860 12825 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:20.940099 12825 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:20.943202 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:20.943490 16157 raft_consensus.cc:3037] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:20.945005 12825 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 7efda0ff8d8c4e71acd4ea20c2e772f5
I20240501 08:12:20.945848 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:20 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:20.946650 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:20 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:20.946869 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:20 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:20.949990 16157 raft_consensus.cc:2454] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:20.951095 12972 raft_consensus.cc:2454] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:20.951814 15689 leader_election.cc:302] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:20.962978 17120 raft_consensus.cc:2781] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:20.963796 17120 raft_consensus.cc:686] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:20.964166 17120 consensus_queue.cc:237] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:20.966315 17244 tablet_replica.cc:330] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:20.966717 17244 raft_consensus.cc:2229] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:20.976681 17244 raft_consensus.cc:2258] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.978452 17244 ts_tablet_manager.cc:1875] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:20.992771 17244 ts_tablet_manager.cc:1888] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:20.993100 17244 log.cc:1196] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/7efda0ff8d8c4e71acd4ea20c2e772f5
I20240501 08:12:20.993645 17242 tablet_replica.cc:330] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:20.993901 17242 raft_consensus.cc:2229] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:20.994060 17244 ts_tablet_manager.cc:1909] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:20.994131 17242 raft_consensus.cc:2258] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.995136 17242 ts_tablet_manager.cc:1875] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:20.995342 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (table rebalancer_dynamic_table_new_name [id=5382759c9bec4cb99561518cbc7c7330]) successfully deleted
I20240501 08:12:20.995626 17243 tablet_replica.cc:330] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:20.995849 17243 raft_consensus.cc:2229] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:20.996349 17243 raft_consensus.cc:2258] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:20.998098 17243 ts_tablet_manager.cc:1875] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:20.999114 12825 catalog_manager.cc:5484] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:21.003638 17077 raft_consensus.cc:2932] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:21.005300 17243 ts_tablet_manager.cc:1888] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.005415 17243 log.cc:1196] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/7efda0ff8d8c4e71acd4ea20c2e772f5
I20240501 08:12:21.006671 17243 ts_tablet_manager.cc:1909] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:21.007926 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (table rebalancer_dynamic_table_new_name [id=5382759c9bec4cb99561518cbc7c7330]) successfully deleted
I20240501 08:12:21.008374 15893 raft_consensus.cc:2932] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:21.009047 17208 raft_consensus.cc:482] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.009220 17208 raft_consensus.cc:504] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:21.010015 17208 leader_election.cc:288] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:21.010308 15755 raft_consensus.cc:2932] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:21.013552 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22fc6797cb24451b78a5c1e5f6306ba" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:21.013808 16157 raft_consensus.cc:2454] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:21.013954 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22fc6797cb24451b78a5c1e5f6306ba" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:21.014182 12974 raft_consensus.cc:2454] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:21.014855 13298 leader_election.cc:302] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:21.015162 17208 raft_consensus.cc:2781] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.015269 17208 raft_consensus.cc:482] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.015357 17208 raft_consensus.cc:3037] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.018064 17208 raft_consensus.cc:504] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:21.019053 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22fc6797cb24451b78a5c1e5f6306ba" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:21.019286 12974 raft_consensus.cc:3037] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.019634 12825 catalog_manager.cc:5484] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:21.020982 17208 leader_election.cc:288] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:21.021327 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f22fc6797cb24451b78a5c1e5f6306ba" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:21.021559 16157 raft_consensus.cc:3037] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.022022 12974 raft_consensus.cc:2454] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:21.022465 13301 leader_election.cc:302] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:21.022819 17208 raft_consensus.cc:2781] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.022943 17208 raft_consensus.cc:686] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:21.023222 17208 consensus_queue.cc:237] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:21.024137 16157 raft_consensus.cc:2454] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:21.034323 17242 ts_tablet_manager.cc:1888] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.035778 17242 log.cc:1196] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/7efda0ff8d8c4e71acd4ea20c2e772f5
I20240501 08:12:21.036258 17242 ts_tablet_manager.cc:1909] T 7efda0ff8d8c4e71acd4ea20c2e772f5 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:21.037379 17091 tablet_bootstrap.cc:492] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:21.037427 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 7efda0ff8d8c4e71acd4ea20c2e772f5 (table rebalancer_dynamic_table_new_name [id=5382759c9bec4cb99561518cbc7c7330]) successfully deleted
I20240501 08:12:21.037827 17091 tablet_bootstrap.cc:492] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:12:21.038084 17091 ts_tablet_manager.cc:1367] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.362s user 0.053s sys 0.003s
I20240501 08:12:21.038821 17091 raft_consensus.cc:348] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.039080 17091 raft_consensus.cc:729] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.039321 17091 consensus_queue.cc:260] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.039861 17091 ts_tablet_manager.cc:1398] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:21.041626 13255 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-7883772f69cd4a18a2a6d568adbdeb61 received from {username='slave'} at 127.6.56.72:54825
I20240501 08:12:21.041759 13255 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-7883772f69cd4a18a2a6d568adbdeb61 on tablet 7883772f69cd4a18a2a6d568adbdeb61 with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:21.044438 16157 raft_consensus.cc:1203] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:21.047796 12828 catalog_manager.cc:5484] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:21.050268 12825 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:21.058036 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 9190731d934f495a8d48bda71fab9083 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=383a883307e74ab39900ca6ceb1f1607]), partition=
I20240501 08:12:21.058205 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 9190731d934f495a8d48bda71fab9083 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=383a883307e74ab39900ca6ceb1f1607]), partition=
I20240501 08:12:21.058276 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 9190731d934f495a8d48bda71fab9083 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=383a883307e74ab39900ca6ceb1f1607]), partition=
I20240501 08:12:21.058429 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9190731d934f495a8d48bda71fab9083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.058569 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9190731d934f495a8d48bda71fab9083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.058596 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9190731d934f495a8d48bda71fab9083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.065207 17212 tablet_bootstrap.cc:492] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:21.066176 17213 tablet_bootstrap.cc:492] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:21.067004 17212 tablet_bootstrap.cc:654] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.067687 17213 tablet_bootstrap.cc:654] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.068074 17209 tablet_bootstrap.cc:492] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:21.069859 17209 tablet_bootstrap.cc:654] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.079823 17213 tablet_bootstrap.cc:492] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:21.080152 17213 ts_tablet_manager.cc:1367] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20240501 08:12:21.081212 17213 raft_consensus.cc:348] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.081626 17213 raft_consensus.cc:374] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.081753 17213 raft_consensus.cc:729] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.082126 17213 consensus_queue.cc:260] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.083840 17213 ts_tablet_manager.cc:1398] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:21.096828 17209 tablet_bootstrap.cc:492] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:21.097030 17209 ts_tablet_manager.cc:1367] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.004s
I20240501 08:12:21.097872 17209 raft_consensus.cc:348] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.098095 17209 raft_consensus.cc:374] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.098186 17209 raft_consensus.cc:729] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.100526 17209 consensus_queue.cc:260] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.101276 17255 ts_tablet_manager.cc:902] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:21.103395 17209 ts_tablet_manager.cc:1398] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 08:12:21.104570 17107 tablet_bootstrap.cc:492] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:21.104840 17212 tablet_bootstrap.cc:492] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:21.105063 17107 tablet_bootstrap.cc:492] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:12:21.105065 17212 ts_tablet_manager.cc:1367] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.040s user 0.013s sys 0.000s
I20240501 08:12:21.105322 17107 ts_tablet_manager.cc:1367] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.225s user 0.062s sys 0.046s
I20240501 08:12:21.106343 17107 raft_consensus.cc:348] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.106544 17107 raft_consensus.cc:729] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.107002 17107 consensus_queue.cc:260] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.107275 17212 raft_consensus.cc:348] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.107494 17212 raft_consensus.cc:374] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.107596 17212 raft_consensus.cc:729] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.107654 17107 ts_tablet_manager.cc:1398] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:21.107857 17212 consensus_queue.cc:260] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.109791 17212 ts_tablet_manager.cc:1398] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:12:21.110949 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-7c4b83a76d7b44039ebb17245fcd8a3c received from {username='slave'} at 127.6.56.71:33711
I20240501 08:12:21.111078 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 5dd722c770d34d6c9154a87917117161-7c4b83a76d7b44039ebb17245fcd8a3c on tablet 7c4b83a76d7b44039ebb17245fcd8a3c with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:12:21.121877 17255 tablet_copy_client.cc:308] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:12:21.122406 16745 raft_consensus.cc:482] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.122572 16745 raft_consensus.cc:504] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.122984 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 8a48ba4666c14848b2b5a1e62f16357b from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:56345)
I20240501 08:12:21.123174 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 8a48ba4666c14848b2b5a1e62f16357b from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:56345: session id = 6ed448d04e4646658c0de37d829ff15b-8a48ba4666c14848b2b5a1e62f16357b
I20240501 08:12:21.123296 16745 leader_election.cc:288] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:21.126809 16045 tablet_copy_source_session.cc:205] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:21.127800 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9190731d934f495a8d48bda71fab9083" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:21.127938 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9190731d934f495a8d48bda71fab9083" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:21.128033 15893 raft_consensus.cc:2454] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:21.128207 13103 raft_consensus.cc:2454] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:21.128648 16090 leader_election.cc:302] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:21.129105 16745 raft_consensus.cc:2781] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.129227 16745 raft_consensus.cc:482] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.129318 16745 raft_consensus.cc:3037] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.132597 16745 raft_consensus.cc:504] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.133562 17255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a48ba4666c14848b2b5a1e62f16357b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.133817 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9190731d934f495a8d48bda71fab9083" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:21.133862 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9190731d934f495a8d48bda71fab9083" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:21.134052 13103 raft_consensus.cc:3037] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.134099 15893 raft_consensus.cc:3037] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.136840 13103 raft_consensus.cc:2454] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:21.137650 16092 leader_election.cc:302] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:21.140398 17255 tablet_copy_client.cc:783] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:21.140693 17255 tablet_copy_client.cc:647] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:21.140770 15893 raft_consensus.cc:2454] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:21.140928 16745 leader_election.cc:288] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:21.141146 16745 raft_consensus.cc:2781] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.141294 16745 raft_consensus.cc:686] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:21.141605 16745 consensus_queue.cc:237] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.144207 17255 tablet_copy_client.cc:523] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:21.147888 16025 raft_consensus.cc:1203] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:21.150359 17255 tablet_bootstrap.cc:492] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:21.152171 12825 catalog_manager.cc:5484] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:21.157658 17255 tablet_bootstrap.cc:492] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:21.158064 17255 tablet_bootstrap.cc:492] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:12:21.158322 17255 ts_tablet_manager.cc:1367] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240501 08:12:21.158998 17255 raft_consensus.cc:348] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:21.159189 17255 raft_consensus.cc:729] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.159435 17255 consensus_queue.cc:260] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:21.159942 17255 ts_tablet_manager.cc:1398] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:21.161198 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-8a48ba4666c14848b2b5a1e62f16357b received from {username='slave'} at 127.6.56.67:56345
I20240501 08:12:21.161368 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-8a48ba4666c14848b2b5a1e62f16357b on tablet 8a48ba4666c14848b2b5a1e62f16357b with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:21.168526 17120 raft_consensus.cc:482] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.168732 17120 raft_consensus.cc:504] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:21.169404 17120 leader_election.cc:288] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:21.169968 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b77e87381d402ab836985100ccb279" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:21.170197 15893 raft_consensus.cc:2454] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:21.170305 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b77e87381d402ab836985100ccb279" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:21.170502 13103 raft_consensus.cc:2454] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:21.170841 15689 leader_election.cc:302] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:21.171108 17120 raft_consensus.cc:2781] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.171211 17120 raft_consensus.cc:482] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.171316 17120 raft_consensus.cc:3037] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.173713 17120 raft_consensus.cc:504] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:21.174360 17120 leader_election.cc:288] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:21.174829 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b77e87381d402ab836985100ccb279" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:21.175038 13103 raft_consensus.cc:3037] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.176000 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55b77e87381d402ab836985100ccb279" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:21.176203 15893 raft_consensus.cc:3037] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.177851 13103 raft_consensus.cc:2454] T 55b77e87381d402ab836985100ccb279 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:21.178354 15691 leader_election.cc:302] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:21.178675 17120 raft_consensus.cc:2781] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.179018 15893 raft_consensus.cc:2454] T 55b77e87381d402ab836985100ccb279 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:21.179236 17120 raft_consensus.cc:686] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:21.179548 17120 consensus_queue.cc:237] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:21.182508 12825 catalog_manager.cc:5484] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:21.272367 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:21.293030 15893 raft_consensus.cc:1261] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:21.293251 13103 raft_consensus.cc:1261] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:21.293886 17262 consensus_queue.cc:1035] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.294196 16745 consensus_queue.cc:1035] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.301374 17169 tablet.cc:1722] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:21.301515 12826 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 9190731d934f495a8d48bda71fab9083 (table rebalancer_dynamic_table [id=383a883307e74ab39900ca6ceb1f1607]). Expected version 1 got 0
I20240501 08:12:21.301945 17169 tablet.cc:1620] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:21.302690 17296 tablet.cc:1722] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:21.303390 17296 tablet.cc:1620] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:21.310545 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=383a883307e74ab39900ca6ceb1f1607] alter complete (version 1)
I20240501 08:12:21.323084 12826 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet 9190731d934f495a8d48bda71fab9083 (table rebalancer_dynamic_table [id=383a883307e74ab39900ca6ceb1f1607]). Expected version 1 got 0
I20240501 08:12:21.325346 17174 tablet.cc:1722] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:21.325901 17174 tablet.cc:1620] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:21.357187 13345 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:21.361953 16004 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:21.381352 15872 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:21.403116 17120 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.405139 16137 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:12:21.418174 12824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:21.436399 12953 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:21.442026 17207 consensus_queue.cc:1035] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.448463 17296 tablet.cc:1722] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:21.449079 17296 tablet.cc:1620] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:21.451875 17169 tablet.cc:1722] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:21.452494 17169 tablet.cc:1620] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:21.455976 17174 tablet.cc:1722] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:21.456558 17174 tablet.cc:1620] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:21.468425 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=383a883307e74ab39900ca6ceb1f1607] alter complete (version 2)
I20240501 08:12:21.484565 13215 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:21.491904 13235 raft_consensus.cc:1203] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:21.501798 17316 ts_tablet_manager.cc:902] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:21.522167 17207 consensus_queue.cc:1035] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.540557 13084 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:21.574410 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:21.588398 17202 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.594100 17296 tablet.cc:1722] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:21.596226 17169 tablet.cc:1722] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:21.606601 17174 tablet.cc:1722] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:21.614356 17120 consensus_queue.cc:1035] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.617421 12828 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=383a883307e74ab39900ca6ceb1f1607] alter complete (version 3)
I20240501 08:12:21.635819 15736 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:12:21.637771 17316 tablet_copy_client.cc:308] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:12:21.640456 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet bdabd94a46dc4a5796949fdf9d0c698c from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:52501)
I20240501 08:12:21.640630 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet bdabd94a46dc4a5796949fdf9d0c698c from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:52501: session id = 76d830e199eb461c8ca689c9460ee0c2-bdabd94a46dc4a5796949fdf9d0c698c
I20240501 08:12:21.643465 16045 tablet_copy_source_session.cc:205] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:21.645042 17202 consensus_queue.cc:1035] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:21.648734 17316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdabd94a46dc4a5796949fdf9d0c698c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.659096 17316 tablet_copy_client.cc:783] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:21.659601 17316 tablet_copy_client.cc:647] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:21.664772 17316 tablet_copy_client.cc:523] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:21.667768 17316 tablet_bootstrap.cc:492] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:21.674572 17316 tablet_bootstrap.cc:492] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:21.674978 17316 tablet_bootstrap.cc:492] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:12:21.675217 17316 ts_tablet_manager.cc:1367] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20240501 08:12:21.675853 17316 raft_consensus.cc:348] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:21.676033 17316 raft_consensus.cc:729] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.676348 17316 consensus_queue.cc:260] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:21.676894 17316 ts_tablet_manager.cc:1398] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 08:12:21.677803 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-bdabd94a46dc4a5796949fdf9d0c698c received from {username='slave'} at 127.6.56.65:52501
I20240501 08:12:21.677953 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-bdabd94a46dc4a5796949fdf9d0c698c on tablet bdabd94a46dc4a5796949fdf9d0c698c with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:21.719966 12826 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:21.720218 12826 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:21.723299 12826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 9190731d934f495a8d48bda71fab9083
I20240501 08:12:21.724529 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 9190731d934f495a8d48bda71fab9083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:21 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:21.724774 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet 9190731d934f495a8d48bda71fab9083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:21 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:21.725163 16137 tablet_service.cc:1514] Processing DeleteTablet for tablet 9190731d934f495a8d48bda71fab9083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:21 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:21.725462 17333 tablet_replica.cc:330] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:21.725644 17334 tablet_replica.cc:330] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:21.725672 17333 raft_consensus.cc:2229] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:21.726063 17334 raft_consensus.cc:2229] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:21.726318 17334 raft_consensus.cc:2258] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:21.727234 17334 ts_tablet_manager.cc:1875] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:21.728597 17333 raft_consensus.cc:2258] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:21.729672 17333 ts_tablet_manager.cc:1875] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:21.733734 17334 ts_tablet_manager.cc:1888] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.733886 17334 log.cc:1196] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/9190731d934f495a8d48bda71fab9083
I20240501 08:12:21.734355 17334 ts_tablet_manager.cc:1909] T 9190731d934f495a8d48bda71fab9083 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:21.735392 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 9190731d934f495a8d48bda71fab9083 (table rebalancer_dynamic_table_new_name [id=383a883307e74ab39900ca6ceb1f1607]) successfully deleted
I20240501 08:12:21.739277 17335 tablet_replica.cc:330] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:21.739548 17335 raft_consensus.cc:2229] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:21.739692 17333 ts_tablet_manager.cc:1888] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.739818 17333 log.cc:1196] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/9190731d934f495a8d48bda71fab9083
I20240501 08:12:21.740381 17333 ts_tablet_manager.cc:1909] T 9190731d934f495a8d48bda71fab9083 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:21.741925 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 9190731d934f495a8d48bda71fab9083 (table rebalancer_dynamic_table_new_name [id=383a883307e74ab39900ca6ceb1f1607]) successfully deleted
I20240501 08:12:21.742684 17335 raft_consensus.cc:2258] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:21.743734 17335 ts_tablet_manager.cc:1875] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:21.749807 17335 ts_tablet_manager.cc:1888] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:21.750385 17335 log.cc:1196] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/9190731d934f495a8d48bda71fab9083
I20240501 08:12:21.750790 17335 ts_tablet_manager.cc:1909] T 9190731d934f495a8d48bda71fab9083 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:21.751761 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 9190731d934f495a8d48bda71fab9083 (table rebalancer_dynamic_table_new_name [id=383a883307e74ab39900ca6ceb1f1607]) successfully deleted
I20240501 08:12:21.771977 16157 consensus_queue.cc:237] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:21.774618 13235 raft_consensus.cc:1261] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:21.774794 13103 raft_consensus.cc:1261] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:21.775169 16092 consensus_peers.cc:485] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c2c8548c31343ff9426fe7eda194703. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:21.775537 16745 consensus_queue.cc:1035] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:21.775874 17262 consensus_queue.cc:1035] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:21.779501 17262 raft_consensus.cc:2932] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:21.780532 13103 raft_consensus.cc:2932] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:21.788344 12826 catalog_manager.cc:5484] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:21.791734 13235 raft_consensus.cc:2932] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:21.796754 12828 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:21.809720 15872 tablet_service.cc:1467] Processing CreateTablet for tablet f8203d209d624d2695e5b32bbfa0edf7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.809880 15736 tablet_service.cc:1467] Processing CreateTablet for tablet f8203d209d624d2695e5b32bbfa0edf7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.810331 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8203d209d624d2695e5b32bbfa0edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.810302 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8203d209d624d2695e5b32bbfa0edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.814078 16004 tablet_service.cc:1467] Processing CreateTablet for tablet f8203d209d624d2695e5b32bbfa0edf7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.814553 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8203d209d624d2695e5b32bbfa0edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.816382 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 03895995b2e5483998d442855006b24c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.816655 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 03895995b2e5483998d442855006b24c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.816776 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03895995b2e5483998d442855006b24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.817063 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03895995b2e5483998d442855006b24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.820727 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 4e330f6ac7074134ba06effe5ececcde (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.821110 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e330f6ac7074134ba06effe5ececcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.826344 12826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:21.827535 17363 tablet_bootstrap.cc:492] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:21.828336 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 4e330f6ac7074134ba06effe5ececcde (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.828768 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e330f6ac7074134ba06effe5ececcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.834862 15871 tablet_service.cc:1467] Processing CreateTablet for tablet 4e330f6ac7074134ba06effe5ececcde (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.834856 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 03895995b2e5483998d442855006b24c (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:21.835310 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03895995b2e5483998d442855006b24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.835319 15871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e330f6ac7074134ba06effe5ececcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.847537 17366 tablet_bootstrap.cc:492] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:21.852070 17366 tablet_bootstrap.cc:654] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.850924 17363 tablet_bootstrap.cc:654] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.878806 17363 tablet_bootstrap.cc:492] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:21.879227 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 50db67986e0249fbbf3bf329e236acd4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b4083e8102348f995632ddd908bd816]), partition=
I20240501 08:12:21.879746 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50db67986e0249fbbf3bf329e236acd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.884771 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 50db67986e0249fbbf3bf329e236acd4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b4083e8102348f995632ddd908bd816]), partition=
I20240501 08:12:21.885182 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50db67986e0249fbbf3bf329e236acd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.889003 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 50db67986e0249fbbf3bf329e236acd4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9b4083e8102348f995632ddd908bd816]), partition=
I20240501 08:12:21.889498 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50db67986e0249fbbf3bf329e236acd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:21.890405 17369 tablet_bootstrap.cc:492] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:21.892796 17370 tablet_bootstrap.cc:492] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:21.895236 17370 tablet_bootstrap.cc:654] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.895452 17369 tablet_bootstrap.cc:654] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.903791 17363 ts_tablet_manager.cc:1367] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.076s user 0.002s sys 0.006s
I20240501 08:12:21.904799 17363 raft_consensus.cc:348] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.905078 17363 raft_consensus.cc:374] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.905164 17363 raft_consensus.cc:729] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.905442 17363 consensus_queue.cc:260] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.906168 17363 ts_tablet_manager.cc:1398] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:21.906514 17363 tablet_bootstrap.cc:492] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:21.913653 17366 tablet_bootstrap.cc:492] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:21.913839 17366 ts_tablet_manager.cc:1367] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.066s user 0.007s sys 0.000s
I20240501 08:12:21.914659 17366 raft_consensus.cc:348] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.914880 17366 raft_consensus.cc:374] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.914983 17366 raft_consensus.cc:729] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.915223 17366 consensus_queue.cc:260] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.916204 17366 ts_tablet_manager.cc:1398] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:21.916519 17366 tablet_bootstrap.cc:492] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:21.918556 17366 tablet_bootstrap.cc:654] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.923228 17370 tablet_bootstrap.cc:492] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:21.923492 17370 ts_tablet_manager.cc:1367] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
I20240501 08:12:21.924307 17370 raft_consensus.cc:348] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.926651 17370 raft_consensus.cc:374] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.926856 17370 raft_consensus.cc:729] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.927150 17370 consensus_queue.cc:260] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.928427 17363 tablet_bootstrap.cc:654] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.928979 17370 ts_tablet_manager.cc:1398] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:21.930147 17369 tablet_bootstrap.cc:492] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:21.930385 17369 ts_tablet_manager.cc:1367] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.005s
I20240501 08:12:21.931548 17369 raft_consensus.cc:348] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.931821 17369 raft_consensus.cc:374] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.931973 17369 raft_consensus.cc:729] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.932318 17369 consensus_queue.cc:260] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.932678 17364 tablet_bootstrap.cc:492] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:21.934489 17364 tablet_bootstrap.cc:654] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.939427 17369 ts_tablet_manager.cc:1398] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 08:12:21.940447 17366 tablet_bootstrap.cc:492] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:21.940685 17366 ts_tablet_manager.cc:1367] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240501 08:12:21.941046 17202 raft_consensus.cc:482] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.941205 17202 raft_consensus.cc:504] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.941550 17366 raft_consensus.cc:348] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.941766 17366 raft_consensus.cc:374] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.941838 17366 raft_consensus.cc:729] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.941941 17202 leader_election.cc:288] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:21.942085 17366 consensus_queue.cc:260] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:21.942715 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8203d209d624d2695e5b32bbfa0edf7" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:21.942867 17366 ts_tablet_manager.cc:1398] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:21.944489 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8203d209d624d2695e5b32bbfa0edf7" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
W20240501 08:12:21.945619 15689 leader_election.cc:341] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:12:21.946244 15691 leader_election.cc:341] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:21.946380 15691 leader_election.cc:302] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161
I20240501 08:12:21.950567 17156 raft_consensus.cc:482] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:21.950764 17156 raft_consensus.cc:504] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.951628 17156 leader_election.cc:288] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:21.953234 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03895995b2e5483998d442855006b24c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
W20240501 08:12:21.955647 15959 leader_election.cc:341] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:21.955931 17202 raft_consensus.cc:2726] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:21.958339 17369 tablet_bootstrap.cc:492] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:21.952545 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03895995b2e5483998d442855006b24c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:21.958622 16157 raft_consensus.cc:2454] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:21.961206 17370 tablet_bootstrap.cc:492] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:21.959295 15957 leader_election.cc:302] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:21.961732 17156 raft_consensus.cc:2781] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:21.961834 17156 raft_consensus.cc:482] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:21.961934 17156 raft_consensus.cc:3037] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.965929 17156 raft_consensus.cc:504] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.967082 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03895995b2e5483998d442855006b24c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
W20240501 08:12:21.967715 15959 leader_election.cc:341] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:21.969012 17156 leader_election.cc:288] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:21.968775 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03895995b2e5483998d442855006b24c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:21.973469 16157 raft_consensus.cc:3037] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:21.977151 16157 raft_consensus.cc:2454] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:21.977764 15957 leader_election.cc:302] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:21.980829 17156 raft_consensus.cc:2781] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:21.990213 17369 tablet_bootstrap.cc:654] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.991493 17370 tablet_bootstrap.cc:654] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:21.996954 17156 raft_consensus.cc:686] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:21.997218 17363 tablet_bootstrap.cc:492] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:21.997454 17363 ts_tablet_manager.cc:1367] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.015s
I20240501 08:12:21.997360 17156 consensus_queue.cc:237] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.998382 17363 raft_consensus.cc:348] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.998677 17363 raft_consensus.cc:374] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:21.998801 17363 raft_consensus.cc:729] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:21.999125 17363 consensus_queue.cc:260] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:21.999765 17363 ts_tablet_manager.cc:1398] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:22.000131 17363 tablet_bootstrap.cc:492] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:22.001685 17363 tablet_bootstrap.cc:654] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.003902 12973 raft_consensus.cc:1203] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:22.007329 17369 tablet_bootstrap.cc:492] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:22.007704 17369 ts_tablet_manager.cc:1367] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.002s
I20240501 08:12:22.010573 17363 tablet_bootstrap.cc:492] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:22.011096 17363 ts_tablet_manager.cc:1367] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240501 08:12:22.012187 17363 raft_consensus.cc:348] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:22.013449 17363 raft_consensus.cc:374] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.013731 17363 raft_consensus.cc:729] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.014297 17363 consensus_queue.cc:260] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:22.017307 17363 ts_tablet_manager.cc:1398] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:22.035988 17364 tablet_bootstrap.cc:492] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:22.036269 17364 ts_tablet_manager.cc:1367] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.104s user 0.009s sys 0.031s
I20240501 08:12:22.037151 17364 raft_consensus.cc:348] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.037458 17364 raft_consensus.cc:374] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.037590 17364 raft_consensus.cc:729] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.037909 17364 consensus_queue.cc:260] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.040571 17364 ts_tablet_manager.cc:1398] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:22.041296 17364 tablet_bootstrap.cc:492] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:22.043033 12826 catalog_manager.cc:5484] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:22.047187 17364 tablet_bootstrap.cc:654] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.050928 17369 raft_consensus.cc:348] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.053959 17369 raft_consensus.cc:374] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.054566 17369 raft_consensus.cc:729] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.054937 17369 consensus_queue.cc:260] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.055219 17364 tablet_bootstrap.cc:492] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:22.055408 17364 ts_tablet_manager.cc:1367] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20240501 08:12:22.056154 17364 raft_consensus.cc:348] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.056443 17364 raft_consensus.cc:374] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.056560 17364 raft_consensus.cc:729] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.056867 17364 consensus_queue.cc:260] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.058753 17364 ts_tablet_manager.cc:1398] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:22.059199 17370 tablet_bootstrap.cc:492] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:22.059405 17370 ts_tablet_manager.cc:1367] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20240501 08:12:22.060070 17370 raft_consensus.cc:348] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:22.060261 17370 raft_consensus.cc:374] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.060351 17370 raft_consensus.cc:729] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.060622 17370 consensus_queue.cc:260] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:22.061196 17370 ts_tablet_manager.cc:1398] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:22.058025 17369 ts_tablet_manager.cc:1398] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20240501 08:12:22.068985 17369 tablet_bootstrap.cc:492] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:22.074100 17369 tablet_bootstrap.cc:654] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.077514 17369 tablet_bootstrap.cc:492] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:22.078159 17369 ts_tablet_manager.cc:1367] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240501 08:12:22.079051 17369 raft_consensus.cc:348] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.079325 17369 raft_consensus.cc:374] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.079447 17369 raft_consensus.cc:729] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.079732 17369 consensus_queue.cc:260] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.081329 17369 ts_tablet_manager.cc:1398] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:12:22.092155 13235 consensus_queue.cc:237] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.094509 12973 raft_consensus.cc:1261] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:22.094781 13170 consensus_peers.cc:485] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94e7fb526fbc4ebc94875c00b49ac43d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:22.095300 17218 consensus_queue.cc:1035] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:22.096732 13103 raft_consensus.cc:1261] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:22.098255 16793 consensus_queue.cc:1035] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:22.099922 17202 raft_consensus.cc:482] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.100090 17202 raft_consensus.cc:504] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:22.100864 17202 leader_election.cc:288] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:22.101294 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50db67986e0249fbbf3bf329e236acd4" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:22.101356 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50db67986e0249fbbf3bf329e236acd4" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:22.101542 13364 raft_consensus.cc:2454] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:22.101617 16157 raft_consensus.cc:2454] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:22.102017 15689 leader_election.cc:302] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:22.102296 17202 raft_consensus.cc:2781] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.102430 17202 raft_consensus.cc:482] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.102524 17202 raft_consensus.cc:3037] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.102320 17218 raft_consensus.cc:2932] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:22.105377 17202 raft_consensus.cc:504] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:22.106408 17202 leader_election.cc:288] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:22.106379 13104 raft_consensus.cc:2932] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:22.106810 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50db67986e0249fbbf3bf329e236acd4" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:22.107000 16157 raft_consensus.cc:3037] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.106839 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50db67986e0249fbbf3bf329e236acd4" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:22.107131 13364 raft_consensus.cc:3037] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.109297 13364 raft_consensus.cc:2454] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:22.110729 15691 leader_election.cc:302] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:22.111034 17325 raft_consensus.cc:2781] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.111161 17325 raft_consensus.cc:686] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:22.111299 16157 raft_consensus.cc:2454] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:22.111467 17325 consensus_queue.cc:237] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:22.112195 12826 catalog_manager.cc:5484] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:22.118005 12973 raft_consensus.cc:2932] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:22.122429 12826 catalog_manager.cc:5484] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:22.138805 17325 raft_consensus.cc:482] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.139046 17325 raft_consensus.cc:504] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.139765 17325 leader_election.cc:288] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:22.140235 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03895995b2e5483998d442855006b24c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:22.140323 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03895995b2e5483998d442855006b24c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:22.140530 16157 raft_consensus.cc:2379] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 982f49fc87f149439cda0de2261b6bfd in current term 1: Already voted for candidate 5dd722c770d34d6c9154a87917117161 in this term.
I20240501 08:12:22.140935 15689 leader_election.cc:302] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:22.141239 17325 raft_consensus.cc:3037] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.143129 17325 raft_consensus.cc:2726] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:22.163512 12973 consensus_queue.cc:237] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:22.166117 13104 raft_consensus.cc:1261] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:22.166776 13364 raft_consensus.cc:1261] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:22.167407 17417 consensus_queue.cc:1035] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:22.167963 17365 consensus_queue.cc:1035] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:22.172237 17417 raft_consensus.cc:2932] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:22.172945 13104 raft_consensus.cc:2932] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
W20240501 08:12:22.175066 12907 consensus_peers.cc:485] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4ba66f68a134d09bc53d840ba8191d6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:22.177942 13364 raft_consensus.cc:2932] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:22.178750 12826 catalog_manager.cc:5484] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:22.222348 16025 consensus_queue.cc:237] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:22.225059 13235 raft_consensus.cc:1261] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:22.225329 15755 raft_consensus.cc:1261] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:22.225574 15957 consensus_peers.cc:485] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82e08735b5db41dcbccec46aa7278530. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:22.225714 17382 consensus_queue.cc:1035] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:22.226050 17381 consensus_queue.cc:1035] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:22.229382 17156 raft_consensus.cc:2932] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:22.229980 13234 raft_consensus.cc:2932] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:22.233768 15755 raft_consensus.cc:2932] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:22.235483 12826 catalog_manager.cc:5484] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:22.268451 13364 consensus_queue.cc:237] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:22.269448 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240501 08:12:22.275595 13300 consensus_peers.cc:485] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74d6cce81d164722afad1935d4695374. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:22.278671 13104 raft_consensus.cc:1261] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:22.281988 17207 consensus_queue.cc:1035] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:22.283249 13234 raft_consensus.cc:1261] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:22.283818 17208 consensus_queue.cc:1035] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:22.285301 16157 raft_consensus.cc:1261] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:22.286041 17202 consensus_queue.cc:1035] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.287815 13364 raft_consensus.cc:1261] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:22.288473 17325 consensus_queue.cc:1035] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.291534 12825 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet 50db67986e0249fbbf3bf329e236acd4 (table rebalancer_dynamic_table [id=9b4083e8102348f995632ddd908bd816]). Expected version 1 got 0
I20240501 08:12:22.297756 17466 tablet.cc:1722] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:22.298291 17466 tablet.cc:1620] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:22.297674 17208 raft_consensus.cc:2932] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:22.300228 12825 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet 50db67986e0249fbbf3bf329e236acd4 (table rebalancer_dynamic_table [id=9b4083e8102348f995632ddd908bd816]). Expected version 1 got 0
I20240501 08:12:22.301666 13235 raft_consensus.cc:2932] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:22.303961 17185 raft_consensus.cc:482] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.304131 17185 raft_consensus.cc:504] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.305079 17185 leader_election.cc:288] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:22.305763 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e330f6ac7074134ba06effe5ececcde" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:22.306030 16025 raft_consensus.cc:2454] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:22.306524 15825 leader_election.cc:302] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:22.307315 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e330f6ac7074134ba06effe5ececcde" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:22.307525 13363 raft_consensus.cc:2454] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:22.308507 13104 raft_consensus.cc:2932] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:22.314160 12828 catalog_manager.cc:5484] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:22.316761 17465 tablet.cc:1722] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:22.317695 17465 tablet.cc:1620] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:22.327126 12827 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet 50db67986e0249fbbf3bf329e236acd4 (table rebalancer_dynamic_table [id=9b4083e8102348f995632ddd908bd816]). Expected version 1 got 0
I20240501 08:12:22.327896 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=9b4083e8102348f995632ddd908bd816] alter complete (version 1)
I20240501 08:12:22.329223 17185 raft_consensus.cc:2781] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.329375 17185 raft_consensus.cc:482] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.329465 17185 raft_consensus.cc:3037] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.331710 17185 raft_consensus.cc:504] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.332278 17469 tablet.cc:1722] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:22.333292 17469 tablet.cc:1620] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:22.333424 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e330f6ac7074134ba06effe5ececcde" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:22.333654 13363 raft_consensus.cc:3037] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.336827 12828 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet 50db67986e0249fbbf3bf329e236acd4 (table rebalancer_dynamic_table [id=9b4083e8102348f995632ddd908bd816]). Expected version 1 got 0
I20240501 08:12:22.338053 13363 raft_consensus.cc:2454] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:22.339303 15825 leader_election.cc:302] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters:
I20240501 08:12:22.341385 17471 raft_consensus.cc:2781] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.341593 17471 raft_consensus.cc:686] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:22.342000 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e330f6ac7074134ba06effe5ececcde" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:22.342262 16025 raft_consensus.cc:3037] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.342046 17471 consensus_queue.cc:237] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.345366 16025 raft_consensus.cc:2454] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:22.346530 17185 leader_election.cc:288] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:22.356683 17467 raft_consensus.cc:482] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:22.357136 17467 raft_consensus.cc:504] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.358446 17467 leader_election.cc:288] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:22.359660 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8203d209d624d2695e5b32bbfa0edf7" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:22.359866 16025 raft_consensus.cc:2454] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:22.359243 17470 ts_tablet_manager.cc:902] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:22.360348 15825 leader_election.cc:302] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:22.361565 17185 raft_consensus.cc:2781] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:22.361712 17185 raft_consensus.cc:482] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:22.361833 17185 raft_consensus.cc:3037] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.365772 12826 catalog_manager.cc:5484] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:22.365810 17185 raft_consensus.cc:504] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.366950 17185 leader_election.cc:288] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:22.367797 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8203d209d624d2695e5b32bbfa0edf7" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:22.368091 15756 raft_consensus.cc:3037] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.371897 15756 raft_consensus.cc:2454] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:22.373046 15825 leader_election.cc:302] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:22.373313 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8203d209d624d2695e5b32bbfa0edf7" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:22.373540 16025 raft_consensus.cc:3037] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:22.373672 17185 raft_consensus.cc:2781] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:22.373859 17185 raft_consensus.cc:686] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:22.374133 17185 consensus_queue.cc:237] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:22.359450 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8203d209d624d2695e5b32bbfa0edf7" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:22.375697 15755 raft_consensus.cc:2362] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1. Re-sending same reply.
I20240501 08:12:22.379066 16025 raft_consensus.cc:2454] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:22.383232 17470 tablet_copy_client.cc:308] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.72:45383
I20240501 08:12:22.383191 12826 catalog_manager.cc:5484] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:22.385524 16178 tablet_copy_service.cc:140] P 6cc32f6e2d49477f9a23ce6e444039fb: Received BeginTabletCopySession request for tablet 6c2c8548c31343ff9426fe7eda194703 from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:60849)
I20240501 08:12:22.385774 16178 tablet_copy_service.cc:161] P 6cc32f6e2d49477f9a23ce6e444039fb: Beginning new tablet copy session on tablet 6c2c8548c31343ff9426fe7eda194703 from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:60849: session id = 982f49fc87f149439cda0de2261b6bfd-6c2c8548c31343ff9426fe7eda194703
I20240501 08:12:22.389346 16178 tablet_copy_source_session.cc:205] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:22.391177 17470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c2c8548c31343ff9426fe7eda194703. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.397104 17470 tablet_copy_client.cc:783] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:22.397478 17470 tablet_copy_client.cc:647] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:22.399503 17470 tablet_copy_client.cc:523] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:22.402669 17470 tablet_bootstrap.cc:492] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:22.408957 17470 tablet_bootstrap.cc:492] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:22.409415 17470 tablet_bootstrap.cc:492] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:12:22.409682 17470 ts_tablet_manager.cc:1367] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20240501 08:12:22.410506 17470 raft_consensus.cc:348] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:22.410740 17470 raft_consensus.cc:729] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.410993 17470 consensus_queue.cc:260] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:22.411567 17470 ts_tablet_manager.cc:1398] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:22.412532 16178 tablet_copy_service.cc:342] P 6cc32f6e2d49477f9a23ce6e444039fb: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-6c2c8548c31343ff9426fe7eda194703 received from {username='slave'} at 127.6.56.69:60849
I20240501 08:12:22.412714 16178 tablet_copy_service.cc:433] P 6cc32f6e2d49477f9a23ce6e444039fb: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-6c2c8548c31343ff9426fe7eda194703 on tablet 6c2c8548c31343ff9426fe7eda194703 with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:22.441509 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:22.449641 17465 tablet.cc:1722] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:22.450208 17465 tablet.cc:1620] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:22.450242 17469 tablet.cc:1722] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:22.450656 17469 tablet.cc:1620] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:22.450583 17466 tablet.cc:1722] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:22.451002 17466 tablet.cc:1620] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:22.461534 12825 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=9b4083e8102348f995632ddd908bd816] alter complete (version 2)
I20240501 08:12:22.488180 17381 consensus_queue.cc:1035] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.497471 17382 consensus_queue.cc:1035] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.565554 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:22.583132 17465 tablet.cc:1722] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:22.583664 17469 tablet.cc:1722] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:22.585805 17466 tablet.cc:1722] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:22.597992 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=9b4083e8102348f995632ddd908bd816] alter complete (version 3)
I20240501 08:12:22.641472 17529 ts_tablet_manager.cc:902] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:22.644955 17530 ts_tablet_manager.cc:902] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:22.649785 15872 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:22.657128 16137 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:22.697832 13345 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:22.705197 17530 tablet_copy_client.cc:308] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:12:22.705966 13255 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet 94e7fb526fbc4ebc94875c00b49ac43d from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:37937)
I20240501 08:12:22.706166 13255 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet 94e7fb526fbc4ebc94875c00b49ac43d from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:37937: session id = 5dd722c770d34d6c9154a87917117161-94e7fb526fbc4ebc94875c00b49ac43d
I20240501 08:12:22.707870 12824 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:22.708103 12824 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:22.709913 13255 tablet_copy_source_session.cc:205] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:22.711470 17530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e7fb526fbc4ebc94875c00b49ac43d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.711848 12824 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 50db67986e0249fbbf3bf329e236acd4
I20240501 08:12:22.713155 13345 tablet_service.cc:1514] Processing DeleteTablet for tablet 50db67986e0249fbbf3bf329e236acd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:22 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:22.715469 17542 tablet_replica.cc:330] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:22.715587 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 50db67986e0249fbbf3bf329e236acd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:22 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:22.716271 17542 raft_consensus.cc:2229] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:22.716895 17542 raft_consensus.cc:2258] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:22.718248 17542 ts_tablet_manager.cc:1875] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:22.721138 16137 tablet_service.cc:1514] Processing DeleteTablet for tablet 50db67986e0249fbbf3bf329e236acd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:22 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:22.722085 16005 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:22.723996 17530 tablet_copy_client.cc:783] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:22.724373 17530 tablet_copy_client.cc:647] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:22.729686 17530 tablet_copy_client.cc:523] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:22.729802 17470 ts_tablet_manager.cc:902] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:22.729799 17542 ts_tablet_manager.cc:1888] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:22.729928 17542 log.cc:1196] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/50db67986e0249fbbf3bf329e236acd4
I20240501 08:12:22.731834 17470 tablet_copy_client.cc:308] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:12:22.733731 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 74d6cce81d164722afad1935d4695374 from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:46029)
I20240501 08:12:22.733987 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 74d6cce81d164722afad1935d4695374 from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:46029: session id = 982f49fc87f149439cda0de2261b6bfd-74d6cce81d164722afad1935d4695374
I20240501 08:12:22.734938 17542 ts_tablet_manager.cc:1909] T 50db67986e0249fbbf3bf329e236acd4 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:22.735618 17530 tablet_bootstrap.cc:492] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:22.738337 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 50db67986e0249fbbf3bf329e236acd4 (table rebalancer_dynamic_table_new_name [id=9b4083e8102348f995632ddd908bd816]) successfully deleted
I20240501 08:12:22.739180 17185 consensus_queue.cc:1035] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.755182 15755 raft_consensus.cc:1203] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:22.775239 13385 tablet_copy_source_session.cc:205] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:22.781359 12953 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:22.810562 17470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d6cce81d164722afad1935d4695374. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.814286 13215 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:22.819274 15735 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:22.823947 12824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:22.840884 15872 tablet_service.cc:1467] Processing CreateTablet for tablet c189688ee6c245008371256fae584d76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2aa002d262484426b839af093b9389b9]), partition=
I20240501 08:12:22.841451 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c189688ee6c245008371256fae584d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.841670 16005 tablet_service.cc:1467] Processing CreateTablet for tablet c189688ee6c245008371256fae584d76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2aa002d262484426b839af093b9389b9]), partition=
I20240501 08:12:22.842185 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c189688ee6c245008371256fae584d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.845397 15735 tablet_service.cc:1467] Processing CreateTablet for tablet c189688ee6c245008371256fae584d76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2aa002d262484426b839af093b9389b9]), partition=
I20240501 08:12:22.845904 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c189688ee6c245008371256fae584d76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:22.855177 17185 consensus_queue.cc:1035] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.884409 17185 consensus_queue.cc:1035] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:22.910959 17552 tablet_replica.cc:330] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:22.911309 17552 raft_consensus.cc:2229] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:22.911607 17552 raft_consensus.cc:2258] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:22.913334 17555 tablet_bootstrap.cc:492] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:22.926878 17556 tablet_bootstrap.cc:492] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:22.934571 17470 tablet_copy_client.cc:783] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:22.935446 17556 tablet_bootstrap.cc:654] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:22.935523 17470 tablet_copy_client.cc:647] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:22.937043 12824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:22.946075 17556 tablet_bootstrap.cc:492] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:22.946902 17556 ts_tablet_manager.cc:1367] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20240501 08:12:22.948115 17556 raft_consensus.cc:348] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:22.949605 17556 raft_consensus.cc:374] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:22.949909 17556 raft_consensus.cc:729] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:22.950465 17556 consensus_queue.cc:260] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:22.952453 17556 ts_tablet_manager.cc:1398] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:22.966280 17472 consensus_queue.cc:1035] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:22.971300 17554 log.cc:924] Time spent T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240501 08:12:22.991534 17552 ts_tablet_manager.cc:1875] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:22.998992 17555 tablet_bootstrap.cc:654] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
W20240501 08:12:23.007594 15689 consensus_peers.cc:485] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:23.008778 17543 tablet_replica.cc:330] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:23.009042 17543 raft_consensus.cc:2229] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:23.009508 17543 raft_consensus.cc:2258] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:23.010637 17543 ts_tablet_manager.cc:1875] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:23.011485 17558 tablet_bootstrap.cc:492] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:23.014084 17558 tablet_bootstrap.cc:654] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.026329 13084 tablet_service.cc:1467] Processing CreateTablet for tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.026791 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.028637 17529 tablet_copy_client.cc:308] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:12:23.030429 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 89fc6e35197e45acb3f934888784d1aa (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.032027 17543 ts_tablet_manager.cc:1888] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:23.032169 17543 log.cc:1196] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/50db67986e0249fbbf3bf329e236acd4
I20240501 08:12:23.032694 17543 ts_tablet_manager.cc:1909] T 50db67986e0249fbbf3bf329e236acd4 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:23.032642 13215 tablet_service.cc:1467] Processing CreateTablet for tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.033108 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.033846 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 50db67986e0249fbbf3bf329e236acd4 (table rebalancer_dynamic_table_new_name [id=9b4083e8102348f995632ddd908bd816]) successfully deleted
I20240501 08:12:23.035037 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 39b12a1d835d428085d4a24233d78c38 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.036599 15871 tablet_service.cc:1467] Processing CreateTablet for tablet 39b12a1d835d428085d4a24233d78c38 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.037135 15871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b12a1d835d428085d4a24233d78c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.037096 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 89fc6e35197e45acb3f934888784d1aa (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.037496 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fc6e35197e45acb3f934888784d1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.040920 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b12a1d835d428085d4a24233d78c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.043490 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 82e08735b5db41dcbccec46aa7278530 from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:50071)
I20240501 08:12:23.043653 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 82e08735b5db41dcbccec46aa7278530 from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:50071: session id = 6cc32f6e2d49477f9a23ce6e444039fb-82e08735b5db41dcbccec46aa7278530
I20240501 08:12:23.046677 16045 tablet_copy_source_session.cc:205] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:23.051054 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 39b12a1d835d428085d4a24233d78c38 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.051359 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 89fc6e35197e45acb3f934888784d1aa (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.051569 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b12a1d835d428085d4a24233d78c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.051872 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fc6e35197e45acb3f934888784d1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.064805 15872 tablet_service.cc:1467] Processing CreateTablet for tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.065605 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.073529 17568 tablet_bootstrap.cc:492] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:23.075500 17568 tablet_bootstrap.cc:654] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.076361 17567 tablet_bootstrap.cc:492] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:23.078584 17567 tablet_bootstrap.cc:654] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.091132 17567 tablet_bootstrap.cc:492] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:23.091440 17567 ts_tablet_manager.cc:1367] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.000s
I20240501 08:12:23.092962 17470 tablet_copy_client.cc:523] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:23.092821 17567 raft_consensus.cc:348] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.093065 17567 raft_consensus.cc:374] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240501 08:12:23.093508 13300 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.66:35509, user_credentials={real_user=slave}} blocked reactor thread for 70687.2us
I20240501 08:12:23.094288 17568 tablet_bootstrap.cc:492] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:23.094390 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89fc6e35197e45acb3f934888784d1aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.094528 17568 ts_tablet_manager.cc:1367] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20240501 08:12:23.094664 17529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e08735b5db41dcbccec46aa7278530. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.095340 17568 raft_consensus.cc:348] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.095535 17568 raft_consensus.cc:374] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.095618 17568 raft_consensus.cc:729] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.095877 17568 consensus_queue.cc:260] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.096459 17568 ts_tablet_manager.cc:1398] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:23.098440 17568 tablet_bootstrap.cc:492] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:23.100193 17567 raft_consensus.cc:729] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.100632 17567 consensus_queue.cc:260] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.103426 17567 ts_tablet_manager.cc:1398] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 08:12:23.104696 17568 tablet_bootstrap.cc:654] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.104946 17567 tablet_bootstrap.cc:492] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:23.106037 17529 tablet_copy_client.cc:783] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:23.106304 17529 tablet_copy_client.cc:647] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:23.107862 17529 tablet_copy_client.cc:523] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:23.109424 17567 tablet_bootstrap.cc:654] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.113445 17529 tablet_bootstrap.cc:492] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:23.118223 17567 tablet_bootstrap.cc:492] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:23.118497 17567 ts_tablet_manager.cc:1367] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20240501 08:12:23.119463 17567 raft_consensus.cc:348] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
W20240501 08:12:23.121098 15825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 147106us
I20240501 08:12:23.126089 17568 tablet_bootstrap.cc:492] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:23.126561 17568 ts_tablet_manager.cc:1367] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.004s
I20240501 08:12:23.129215 17568 raft_consensus.cc:348] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.130561 17568 raft_consensus.cc:374] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.131119 17568 raft_consensus.cc:729] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.131640 17568 consensus_queue.cc:260] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.133967 17567 raft_consensus.cc:374] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.134546 17567 raft_consensus.cc:729] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.135368 17567 consensus_queue.cc:260] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.139585 17555 tablet_bootstrap.cc:492] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:23.144172 17567 ts_tablet_manager.cc:1398] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240501 08:12:23.144404 17555 ts_tablet_manager.cc:1367] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.231s user 0.004s sys 0.000s
I20240501 08:12:23.145645 17555 raft_consensus.cc:348] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.145941 17558 tablet_bootstrap.cc:492] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:23.146173 17558 ts_tablet_manager.cc:1367] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.135s user 0.009s sys 0.004s
I20240501 08:12:23.146951 17558 raft_consensus.cc:348] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.147249 17558 raft_consensus.cc:374] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.147372 17558 raft_consensus.cc:729] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.147681 17558 consensus_queue.cc:260] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.148869 17555 raft_consensus.cc:374] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.149053 17555 raft_consensus.cc:729] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.149119 17558 ts_tablet_manager.cc:1398] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:23.149444 17558 tablet_bootstrap.cc:492] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:23.149392 17555 consensus_queue.cc:260] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.150030 17555 ts_tablet_manager.cc:1398] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:23.151049 17558 tablet_bootstrap.cc:654] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.152408 17555 tablet_bootstrap.cc:492] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:23.155212 17548 ts_tablet_manager.cc:902] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:23.155210 17568 ts_tablet_manager.cc:1398] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.026s user 0.001s sys 0.002s
I20240501 08:12:23.157721 17555 tablet_bootstrap.cc:654] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
W20240501 08:12:23.160647 13040 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 136003us
W20240501 08:12:23.162727 13041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 82454.3us
I20240501 08:12:23.173740 17555 tablet_bootstrap.cc:492] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:23.174038 17555 ts_tablet_manager.cc:1367] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.004s
I20240501 08:12:23.174608 17470 tablet_bootstrap.cc:492] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:23.174961 17555 raft_consensus.cc:348] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.175253 17555 raft_consensus.cc:374] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.175381 17555 raft_consensus.cc:729] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.175719 17555 consensus_queue.cc:260] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.176596 17555 ts_tablet_manager.cc:1398] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:23.176978 17555 tablet_bootstrap.cc:492] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:23.178978 17555 tablet_bootstrap.cc:654] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.184193 13084 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:23.189129 17558 tablet_bootstrap.cc:492] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:23.189419 17558 ts_tablet_manager.cc:1367] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.001s
I20240501 08:12:23.190310 17558 raft_consensus.cc:348] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.191267 17558 raft_consensus.cc:374] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.191418 17558 raft_consensus.cc:729] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.191860 17558 consensus_queue.cc:260] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.193607 17558 ts_tablet_manager.cc:1398] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:23.193966 17558 tablet_bootstrap.cc:492] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:23.195668 17558 tablet_bootstrap.cc:654] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.220403 17562 raft_consensus.cc:482] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.225557 17558 tablet_bootstrap.cc:492] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:23.225822 17558 ts_tablet_manager.cc:1367] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.002s
I20240501 08:12:23.226693 17558 raft_consensus.cc:348] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.228343 17558 raft_consensus.cc:374] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.220631 17562 raft_consensus.cc:504] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.228523 17558 raft_consensus.cc:729] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.230530 17573 tablet_bootstrap.cc:492] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:23.230633 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b12a1d835d428085d4a24233d78c38" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:23.230903 13234 raft_consensus.cc:2454] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:23.231590 15826 leader_election.cc:302] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:23.231913 17185 raft_consensus.cc:2781] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:23.232025 17185 raft_consensus.cc:482] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:23.232110 17185 raft_consensus.cc:3037] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.233060 17573 tablet_bootstrap.cc:654] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:23.233078 17558 consensus_queue.cc:260] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.236900 17558 ts_tablet_manager.cc:1398] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:12:23.229929 17562 leader_election.cc:288] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:23.239192 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b12a1d835d428085d4a24233d78c38" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:23.239429 15755 raft_consensus.cc:2454] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:23.237480 17185 raft_consensus.cc:504] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.241282 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b12a1d835d428085d4a24233d78c38" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:23.241516 15755 raft_consensus.cc:3037] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.244670 15755 raft_consensus.cc:2454] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:23.246805 15825 leader_election.cc:302] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:23.247406 17472 raft_consensus.cc:2781] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:23.248077 17472 raft_consensus.cc:686] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:23.248395 17472 consensus_queue.cc:237] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.250057 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39b12a1d835d428085d4a24233d78c38" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:23.250299 13234 raft_consensus.cc:3037] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.255020 13234 raft_consensus.cc:2454] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:23.255738 17185 leader_election.cc:288] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:23.264719 17529 tablet_bootstrap.cc:492] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:23.268087 17529 tablet_bootstrap.cc:492] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:12:23.270045 12826 catalog_manager.cc:5484] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:23.271767 17529 ts_tablet_manager.cc:1367] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.155s user 0.003s sys 0.010s
I20240501 08:12:23.273530 17529 raft_consensus.cc:348] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:23.273757 17529 raft_consensus.cc:729] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.274116 17529 consensus_queue.cc:260] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:23.274991 17529 ts_tablet_manager.cc:1398] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:23.276419 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-82e08735b5db41dcbccec46aa7278530 received from {username='slave'} at 127.6.56.72:50071
I20240501 08:12:23.276568 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-82e08735b5db41dcbccec46aa7278530 on tablet 82e08735b5db41dcbccec46aa7278530 with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:23.305243 17548 tablet_copy_client.cc:308] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:12:23.306437 17202 raft_consensus.cc:482] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.306634 17202 raft_consensus.cc:504] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.307335 17202 leader_election.cc:288] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:23.308652 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc6e35197e45acb3f934888784d1aa" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:23.308924 13104 raft_consensus.cc:2454] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:23.309976 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet d4ba66f68a134d09bc53d840ba8191d6 from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:38689)
I20240501 08:12:23.310142 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet d4ba66f68a134d09bc53d840ba8191d6 from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:38689: session id = 4d516916eeac468db514eeb928f06190-d4ba66f68a134d09bc53d840ba8191d6
I20240501 08:12:23.310129 15691 leader_election.cc:302] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:23.310467 17202 raft_consensus.cc:2781] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:23.310549 17202 raft_consensus.cc:482] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:23.310643 17202 raft_consensus.cc:3037] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.313215 12994 tablet_copy_source_session.cc:205] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:23.313606 17202 raft_consensus.cc:504] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.315948 17548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4ba66f68a134d09bc53d840ba8191d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.317090 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc6e35197e45acb3f934888784d1aa" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:23.317288 13104 raft_consensus.cc:3037] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.320801 13104 raft_consensus.cc:2454] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:23.324167 17548 tablet_copy_client.cc:783] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:23.324545 17548 tablet_copy_client.cc:647] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:23.325484 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc6e35197e45acb3f934888784d1aa" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:23.326035 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc6e35197e45acb3f934888784d1aa" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:23.326865 17548 tablet_copy_client.cc:523] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:23.328894 17552 ts_tablet_manager.cc:1888] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:23.329030 17552 log.cc:1196] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/50db67986e0249fbbf3bf329e236acd4
I20240501 08:12:23.329443 17552 ts_tablet_manager.cc:1909] T 50db67986e0249fbbf3bf329e236acd4 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:23.330871 17472 raft_consensus.cc:482] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.331043 17472 raft_consensus.cc:504] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.331816 17573 tablet_bootstrap.cc:492] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:23.331785 17472 leader_election.cc:288] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:23.331990 17573 ts_tablet_manager.cc:1367] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.000s
I20240501 08:12:23.332664 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c189688ee6c245008371256fae584d76" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:23.332944 16024 raft_consensus.cc:2454] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:23.332930 17573 raft_consensus.cc:348] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.333122 17573 raft_consensus.cc:374] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.333204 17573 raft_consensus.cc:729] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.333464 15825 leader_election.cc:302] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:23.333627 17573 consensus_queue.cc:260] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.334255 17573 ts_tablet_manager.cc:1398] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:23.334427 17548 tablet_bootstrap.cc:492] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:23.335276 17472 raft_consensus.cc:2781] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:23.335388 17472 raft_consensus.cc:482] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:23.335469 17472 raft_consensus.cc:3037] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.338866 17472 raft_consensus.cc:504] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.341780 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c189688ee6c245008371256fae584d76" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:23.342041 16024 raft_consensus.cc:3037] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.346068 16024 raft_consensus.cc:2454] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:23.346652 15825 leader_election.cc:302] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:23.347071 17472 leader_election.cc:288] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:23.348143 17202 leader_election.cc:288] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:23.348511 17555 tablet_bootstrap.cc:492] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:23.348651 17555 ts_tablet_manager.cc:1367] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.172s user 0.016s sys 0.062s
W20240501 08:12:23.349614 15689 leader_election.cc:341] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:23.349732 17555 raft_consensus.cc:348] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.349932 17555 raft_consensus.cc:374] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:23.350018 17555 raft_consensus.cc:729] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.350246 17555 consensus_queue.cc:260] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.353514 17555 ts_tablet_manager.cc:1398] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:23.347043 17185 raft_consensus.cc:2781] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:23.353799 17185 raft_consensus.cc:686] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:23.354043 17185 consensus_queue.cc:237] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.356168 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 50db67986e0249fbbf3bf329e236acd4 (table rebalancer_dynamic_table_new_name [id=9b4083e8102348f995632ddd908bd816]) successfully deleted
I20240501 08:12:23.360328 15691 leader_election.cc:302] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd, e3fefd4dca2d41678187047f06c10187; no voters:
W20240501 08:12:23.362854 15689 leader_election.cc:341] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:23.364164 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c189688ee6c245008371256fae584d76" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:23.364513 15756 raft_consensus.cc:3037] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.366454 12828 catalog_manager.cc:5484] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:23.366843 17202 raft_consensus.cc:2781] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:23.363901 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c189688ee6c245008371256fae584d76" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:23.379560 17530 tablet_bootstrap.cc:492] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:23.380074 17530 tablet_bootstrap.cc:492] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:12:23.380510 17530 ts_tablet_manager.cc:1367] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.645s user 0.049s sys 0.007s
I20240501 08:12:23.381644 17530 raft_consensus.cc:348] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:23.381981 17530 raft_consensus.cc:729] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.382408 17530 consensus_queue.cc:260] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:23.383320 17530 ts_tablet_manager.cc:1398] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:23.384187 13255 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-94e7fb526fbc4ebc94875c00b49ac43d received from {username='slave'} at 127.6.56.71:37937
I20240501 08:12:23.384388 13255 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 5dd722c770d34d6c9154a87917117161-94e7fb526fbc4ebc94875c00b49ac43d on tablet 94e7fb526fbc4ebc94875c00b49ac43d with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:12:23.384420 17577 raft_consensus.cc:482] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.384586 17577 raft_consensus.cc:504] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.385529 17577 leader_election.cc:288] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:23.386180 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fda95d9e9d48e7a4d6eb4d9b85fb57" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:23.386409 15893 raft_consensus.cc:2454] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:23.386940 13038 leader_election.cc:302] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:23.387953 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fda95d9e9d48e7a4d6eb4d9b85fb57" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:23.388207 13235 raft_consensus.cc:2454] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:23.389194 17318 raft_consensus.cc:2781] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:23.389284 17318 raft_consensus.cc:482] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:23.389369 17318 raft_consensus.cc:3037] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.395074 17318 raft_consensus.cc:504] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.395815 17318 leader_election.cc:288] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:23.396579 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fda95d9e9d48e7a4d6eb4d9b85fb57" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:23.399304 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fda95d9e9d48e7a4d6eb4d9b85fb57" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:23.399634 15893 raft_consensus.cc:3037] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.400504 13235 raft_consensus.cc:3037] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.401381 15756 raft_consensus.cc:2454] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:23.403406 13235 raft_consensus.cc:2454] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:23.403929 17592 raft_consensus.cc:482] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.404129 17592 raft_consensus.cc:504] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.404894 17592 leader_election.cc:288] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:23.406024 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c189688ee6c245008371256fae584d76" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:23.406239 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c189688ee6c245008371256fae584d76" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:23.406486 16024 raft_consensus.cc:2454] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:23.408942 13041 leader_election.cc:302] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:23.409857 15893 raft_consensus.cc:2454] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:23.410991 15691 leader_election.cc:302] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190
I20240501 08:12:23.411556 17592 raft_consensus.cc:2781] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:12:23.411705 17592 raft_consensus.cc:482] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:23.411816 17592 raft_consensus.cc:3037] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:23.412173 17318 raft_consensus.cc:2781] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:23.412364 17318 raft_consensus.cc:686] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:23.412673 17318 consensus_queue.cc:237] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:23.417089 17592 raft_consensus.cc:504] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.419178 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c189688ee6c245008371256fae584d76" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:23.419458 16024 raft_consensus.cc:3037] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:23.423810 16024 raft_consensus.cc:2454] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 2.
I20240501 08:12:23.424489 17202 raft_consensus.cc:686] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:23.424708 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c189688ee6c245008371256fae584d76" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:23.425163 17202 consensus_queue.cc:237] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.425835 15691 leader_election.cc:302] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190
I20240501 08:12:23.426035 17592 leader_election.cc:288] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:23.429222 17593 raft_consensus.cc:2781] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:23.429422 17593 raft_consensus.cc:686] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 2 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:23.429688 17593 consensus_queue.cc:237] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.435305 12828 catalog_manager.cc:5484] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:23.440518 12824 catalog_manager.cc:5484] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:23.446959 12828 catalog_manager.cc:5484] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 1 to 2, leader changed from 4d516916eeac468db514eeb928f06190 (127.6.56.70) to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 2 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:23.481169 17470 tablet_bootstrap.cc:492] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:23.481768 17470 tablet_bootstrap.cc:492] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:12:23.482558 17470 ts_tablet_manager.cc:1367] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.308s user 0.045s sys 0.006s
I20240501 08:12:23.483300 17548 tablet_bootstrap.cc:492] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:23.483518 17470 raft_consensus.cc:348] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:23.483772 17470 raft_consensus.cc:729] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.484014 17548 tablet_bootstrap.cc:492] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:12:23.484333 17548 ts_tablet_manager.cc:1367] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.150s user 0.065s sys 0.023s
I20240501 08:12:23.484135 17470 consensus_queue.cc:260] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:23.484805 17470 ts_tablet_manager.cc:1398] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:23.485239 17548 raft_consensus.cc:348] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.485528 17548 raft_consensus.cc:729] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:23.485535 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-74d6cce81d164722afad1935d4695374 received from {username='slave'} at 127.6.56.69:46029
I20240501 08:12:23.485656 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-74d6cce81d164722afad1935d4695374 on tablet 74d6cce81d164722afad1935d4695374 with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:23.485801 17548 consensus_queue.cc:260] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.486395 17548 ts_tablet_manager.cc:1398] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:23.487398 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-d4ba66f68a134d09bc53d840ba8191d6 received from {username='slave'} at 127.6.56.70:38689
I20240501 08:12:23.487574 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 4d516916eeac468db514eeb928f06190-d4ba66f68a134d09bc53d840ba8191d6 on tablet d4ba66f68a134d09bc53d840ba8191d6 with peer 4d516916eeac468db514eeb928f06190
I20240501 08:12:23.489457 16153 raft_consensus.cc:1203] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:23.552021 12824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:23.558840 15893 raft_consensus.cc:3032] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:23.559008 15893 raft_consensus.cc:729] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:23.559363 15893 consensus_queue.cc:260] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:23.559846 15893 raft_consensus.cc:3037] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:23.559897 16024 raft_consensus.cc:1261] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:12:23.560992 17593 consensus_queue.cc:1035] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.564805 15893 raft_consensus.cc:1261] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 08:12:23.566516 17202 consensus_queue.cc:1035] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.566731 17202 consensus_queue.cc:1225] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Peer 4d516916eeac468db514eeb928f06190 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240501 08:12:23.568239 12824 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet c189688ee6c245008371256fae584d76 (table rebalancer_dynamic_table [id=2aa002d262484426b839af093b9389b9]). Expected version 1 got 0
I20240501 08:12:23.569729 15893 pending_rounds.cc:76] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Aborting all ops after (but not including) 0
I20240501 08:12:23.569826 15893 pending_rounds.cc:97] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240501 08:12:23.573173 15893 raft_consensus.cc:2866] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:12:23.577472 12824 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet c189688ee6c245008371256fae584d76 (table rebalancer_dynamic_table [id=2aa002d262484426b839af093b9389b9]). Expected version 1 got 0
I20240501 08:12:23.578230 17588 raft_consensus.cc:482] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:23.578420 17588 raft_consensus.cc:504] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:23.574115 17613 tablet.cc:1722] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:23.579360 17588 leader_election.cc:288] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:23.579963 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc6e35197e45acb3f934888784d1aa" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:23.580395 17615 tablet.cc:1722] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:23.580487 17613 tablet.cc:1620] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:23.580963 17615 tablet.cc:1620] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:23.583082 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89fc6e35197e45acb3f934888784d1aa" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:23.583370 13104 raft_consensus.cc:2379] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cc32f6e2d49477f9a23ce6e444039fb in current term 1: Already voted for candidate 982f49fc87f149439cda0de2261b6bfd in this term.
I20240501 08:12:23.583887 16092 leader_election.cc:302] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 982f49fc87f149439cda0de2261b6bfd, e3fefd4dca2d41678187047f06c10187
I20240501 08:12:23.584391 17588 raft_consensus.cc:3037] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:23.587174 17588 raft_consensus.cc:2726] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:23.590129 17614 tablet.cc:1722] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:23.592648 12824 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet c189688ee6c245008371256fae584d76 (table rebalancer_dynamic_table [id=2aa002d262484426b839af093b9389b9]). Expected version 1 got 0
I20240501 08:12:23.593621 12828 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet c189688ee6c245008371256fae584d76 (table rebalancer_dynamic_table [id=2aa002d262484426b839af093b9389b9]). Expected version 1 got 0
I20240501 08:12:23.595659 17614 tablet.cc:1620] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:23.603410 12828 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=2aa002d262484426b839af093b9389b9] alter complete (version 1)
I20240501 08:12:23.622838 15893 raft_consensus.cc:1203] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:23.642792 17185 consensus_queue.cc:1035] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.650468 17472 consensus_queue.cc:1035] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.707334 12824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:23.722003 17613 tablet.cc:1722] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:23.722576 17613 tablet.cc:1620] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:23.725535 16024 raft_consensus.cc:1203] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:23.725584 17615 tablet.cc:1722] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:23.726078 17615 tablet.cc:1620] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:23.743268 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=2aa002d262484426b839af093b9389b9] alter complete (version 2)
I20240501 08:12:23.746459 17614 tablet.cc:1722] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:23.746935 17614 tablet.cc:1620] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:23.786739 13345 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:23.791465 15872 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:23.804505 16005 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:23.804944 15756 raft_consensus.cc:1203] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:23.812455 17202 consensus_queue.cc:1035] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.830135 13214 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:23.830313 17577 consensus_queue.cc:1035] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.844820 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:23.858536 17613 tablet.cc:1722] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:23.860818 17615 tablet.cc:1722] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:23.860968 17614 tablet.cc:1722] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:23.868583 16137 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:12:23.884467 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=2aa002d262484426b839af093b9389b9] alter complete (version 3)
I20240501 08:12:23.909806 17652 consensus_queue.cc:1035] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.966274 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240501 08:12:23.969226 13301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 82670.8us
I20240501 08:12:23.989850 12830 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:23.990108 12830 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:23.991415 17318 consensus_queue.cc:1035] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:23.996083 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 9259925ce64e4b93afc8b08045c19cd9 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.996147 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 9259925ce64e4b93afc8b08045c19cd9 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:23.996611 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9259925ce64e4b93afc8b08045c19cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:23.996639 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9259925ce64e4b93afc8b08045c19cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.002806 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 9259925ce64e4b93afc8b08045c19cd9 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.003329 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9259925ce64e4b93afc8b08045c19cd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.008821 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 714a84eae9324caeb4f25e798d6c35c2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.009330 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714a84eae9324caeb4f25e798d6c35c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.014047 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 714a84eae9324caeb4f25e798d6c35c2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.014621 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714a84eae9324caeb4f25e798d6c35c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.014994 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 279b5f4d8fba4c12beec040c6ed39992 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.015444 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279b5f4d8fba4c12beec040c6ed39992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.016764 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 279b5f4d8fba4c12beec040c6ed39992 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.017308 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279b5f4d8fba4c12beec040c6ed39992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.021490 15871 tablet_service.cc:1467] Processing CreateTablet for tablet 279b5f4d8fba4c12beec040c6ed39992 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.021984 15871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279b5f4d8fba4c12beec040c6ed39992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.034960 17679 tablet_bootstrap.cc:492] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:24.036000 12830 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet c189688ee6c245008371256fae584d76
I20240501 08:12:24.037360 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet c189688ee6c245008371256fae584d76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:23 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:24.039073 17679 tablet_bootstrap.cc:654] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.040963 17681 tablet_replica.cc:330] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:24.041265 17681 raft_consensus.cc:2229] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:24.042968 17681 raft_consensus.cc:2258] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.043094 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet c189688ee6c245008371256fae584d76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:23 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:24.047030 17681 ts_tablet_manager.cc:1875] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:24.049276 17684 tablet_bootstrap.cc:492] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:24.051957 17684 tablet_bootstrap.cc:654] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.066980 17686 tablet_bootstrap.cc:492] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
W20240501 08:12:24.080718 13301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 106568us
I20240501 08:12:24.092803 17680 tablet_bootstrap.cc:492] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:24.097522 17683 tablet_replica.cc:330] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:24.097851 17683 raft_consensus.cc:2229] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:24.098256 17683 raft_consensus.cc:2258] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.099496 17683 ts_tablet_manager.cc:1875] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:24.102455 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 714a84eae9324caeb4f25e798d6c35c2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:24.102960 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714a84eae9324caeb4f25e798d6c35c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.104771 17686 tablet_bootstrap.cc:654] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.112685 17684 tablet_bootstrap.cc:492] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:24.113585 17681 ts_tablet_manager.cc:1888] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.4
I20240501 08:12:24.113682 17684 ts_tablet_manager.cc:1367] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.015s
I20240501 08:12:24.116232 17681 log.cc:1196] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/c189688ee6c245008371256fae584d76
I20240501 08:12:24.117353 17681 ts_tablet_manager.cc:1909] T c189688ee6c245008371256fae584d76 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:24.117259 17684 raft_consensus.cc:348] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.118474 17684 raft_consensus.cc:374] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.118644 17684 raft_consensus.cc:729] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.119130 17684 consensus_queue.cc:260] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.120674 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet c189688ee6c245008371256fae584d76 (table rebalancer_dynamic_table_new_name [id=2aa002d262484426b839af093b9389b9]) successfully deleted
I20240501 08:12:24.121085 17684 ts_tablet_manager.cc:1398] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:12:24.121295 17686 tablet_bootstrap.cc:492] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:24.121423 17684 tablet_bootstrap.cc:492] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:24.121517 17686 ts_tablet_manager.cc:1367] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.007s
I20240501 08:12:24.122540 17686 raft_consensus.cc:348] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.122860 17686 raft_consensus.cc:374] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.123021 17686 raft_consensus.cc:729] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.123387 17686 consensus_queue.cc:260] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.126839 17684 tablet_bootstrap.cc:654] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.127337 17686 ts_tablet_manager.cc:1398] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:24.133941 17684 tablet_bootstrap.cc:492] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:24.133996 17685 tablet_bootstrap.cc:492] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:24.134452 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet c189688ee6c245008371256fae584d76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:23 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:24.134620 17684 ts_tablet_manager.cc:1367] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20240501 08:12:24.135586 17684 raft_consensus.cc:348] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.137353 17684 raft_consensus.cc:374] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.137681 17684 raft_consensus.cc:729] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.138291 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:24.138283 17684 consensus_queue.cc:260] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.138661 17685 tablet_bootstrap.cc:654] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.139050 17684 ts_tablet_manager.cc:1398] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:24.140214 17694 tablet_replica.cc:330] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:24.140491 17694 raft_consensus.cc:2229] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 2 LEADER]: Raft consensus shutting down.
I20240501 08:12:24.143030 17694 raft_consensus.cc:2258] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.144098 17694 ts_tablet_manager.cc:1875] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:24.152647 17694 ts_tablet_manager.cc:1888] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.4
I20240501 08:12:24.155012 17680 tablet_bootstrap.cc:654] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.156685 17694 log.cc:1196] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/c189688ee6c245008371256fae584d76
I20240501 08:12:24.152714 17692 tablet_bootstrap.cc:492] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:24.153472 17679 tablet_bootstrap.cc:492] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:24.158298 17679 ts_tablet_manager.cc:1367] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.015s
I20240501 08:12:24.158964 17692 tablet_bootstrap.cc:654] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.160465 17679 raft_consensus.cc:348] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.160802 17679 raft_consensus.cc:374] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.160879 17694 ts_tablet_manager.cc:1909] T c189688ee6c245008371256fae584d76 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:24.160964 17679 raft_consensus.cc:729] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.166221 17692 tablet_bootstrap.cc:492] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:24.166924 17692 ts_tablet_manager.cc:1367] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240501 08:12:24.161742 17679 consensus_queue.cc:260] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.168227 17692 raft_consensus.cc:348] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.169449 17692 raft_consensus.cc:374] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.169706 17692 raft_consensus.cc:729] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.170006 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:24.170092 17692 consensus_queue.cc:260] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.172031 17692 ts_tablet_manager.cc:1398] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:12:24.176380 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 53a8174993804850839b063726d66df5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1442e0ca3ea042b5a8980acede0c0735]), partition=
I20240501 08:12:24.176569 15871 tablet_service.cc:1467] Processing CreateTablet for tablet 53a8174993804850839b063726d66df5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1442e0ca3ea042b5a8980acede0c0735]), partition=
I20240501 08:12:24.176760 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a8174993804850839b063726d66df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.176939 15871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a8174993804850839b063726d66df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.183743 17684 tablet_bootstrap.cc:492] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:24.185286 17684 tablet_bootstrap.cc:654] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.188501 17678 raft_consensus.cc:482] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.188706 17678 raft_consensus.cc:504] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.189474 17678 leader_election.cc:288] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:24.190162 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714a84eae9324caeb4f25e798d6c35c2" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:24.190445 15754 raft_consensus.cc:2454] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:24.191036 12953 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:24.193228 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714a84eae9324caeb4f25e798d6c35c2" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
W20240501 08:12:24.193851 12907 leader_election.cc:341] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:24.196786 12909 leader_election.cc:302] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters: 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:24.198837 17678 raft_consensus.cc:2781] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.198946 17678 raft_consensus.cc:482] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.199021 17678 raft_consensus.cc:3037] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.202991 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet c189688ee6c245008371256fae584d76 (table rebalancer_dynamic_table_new_name [id=2aa002d262484426b839af093b9389b9]) successfully deleted
I20240501 08:12:24.206045 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 53a8174993804850839b063726d66df5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1442e0ca3ea042b5a8980acede0c0735]), partition=
I20240501 08:12:24.206535 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a8174993804850839b063726d66df5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:24.218554 17678 raft_consensus.cc:504] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.235550 17692 tablet_bootstrap.cc:492] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:24.237828 17692 tablet_bootstrap.cc:654] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.241773 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714a84eae9324caeb4f25e798d6c35c2" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:24.242055 17680 tablet_bootstrap.cc:492] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:24.242269 17680 ts_tablet_manager.cc:1367] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.150s user 0.001s sys 0.009s
W20240501 08:12:24.242477 12907 leader_election.cc:341] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:24.243321 17680 raft_consensus.cc:348] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.243604 17680 raft_consensus.cc:374] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.243754 17680 raft_consensus.cc:729] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.244202 17684 tablet_bootstrap.cc:492] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:24.244072 17680 consensus_queue.cc:260] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.244455 17684 ts_tablet_manager.cc:1367] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.024s
I20240501 08:12:24.244771 17678 leader_election.cc:288] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:24.245805 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714a84eae9324caeb4f25e798d6c35c2" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:24.245951 17680 ts_tablet_manager.cc:1398] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:24.246021 15754 raft_consensus.cc:3037] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.246240 17680 tablet_bootstrap.cc:492] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:24.249588 15754 raft_consensus.cc:2454] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:24.250108 12909 leader_election.cc:302] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters: 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:24.250661 17684 raft_consensus.cc:348] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.250804 17678 raft_consensus.cc:2781] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.250993 17678 raft_consensus.cc:686] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:24.251408 17684 raft_consensus.cc:374] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.251397 17678 consensus_queue.cc:237] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.251734 17684 raft_consensus.cc:729] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.252279 17684 consensus_queue.cc:260] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.253517 17684 ts_tablet_manager.cc:1398] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:24.254621 17680 tablet_bootstrap.cc:654] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.256142 17692 tablet_bootstrap.cc:492] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:24.256443 17692 ts_tablet_manager.cc:1367] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.008s
I20240501 08:12:24.257483 17692 raft_consensus.cc:348] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.257818 17692 raft_consensus.cc:374] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.257992 17692 raft_consensus.cc:729] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.258316 17692 consensus_queue.cc:260] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.259967 17692 ts_tablet_manager.cc:1398] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:24.267462 17680 tablet_bootstrap.cc:492] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:24.268113 17680 ts_tablet_manager.cc:1367] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240501 08:12:24.209247 17683 ts_tablet_manager.cc:1888] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.4
I20240501 08:12:24.269196 17185 raft_consensus.cc:482] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.269229 17156 raft_consensus.cc:482] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.269245 17683 log.cc:1196] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/c189688ee6c245008371256fae584d76
I20240501 08:12:24.269407 17185 raft_consensus.cc:504] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.269491 17156 raft_consensus.cc:504] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.270052 17683 ts_tablet_manager.cc:1909] T c189688ee6c245008371256fae584d76 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:24.270259 17185 leader_election.cc:288] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:24.270462 17156 leader_election.cc:288] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:24.271041 13231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279b5f4d8fba4c12beec040c6ed39992" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:24.271178 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279b5f4d8fba4c12beec040c6ed39992" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:24.271458 15893 raft_consensus.cc:2454] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
W20240501 08:12:24.271828 15960 leader_election.cc:341] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:24.272384 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9259925ce64e4b93afc8b08045c19cd9" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:24.272567 17679 ts_tablet_manager.cc:1398] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.114s user 0.000s sys 0.002s
I20240501 08:12:24.272887 17679 tablet_bootstrap.cc:492] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
W20240501 08:12:24.273066 15823 leader_election.cc:341] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:24.273219 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9259925ce64e4b93afc8b08045c19cd9" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:24.273422 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet c189688ee6c245008371256fae584d76 (table rebalancer_dynamic_table_new_name [id=2aa002d262484426b839af093b9389b9]) successfully deleted
I20240501 08:12:24.274506 15957 leader_election.cc:302] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters: 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:24.274855 17680 raft_consensus.cc:348] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.275178 17680 raft_consensus.cc:374] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.275303 17680 raft_consensus.cc:729] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
W20240501 08:12:24.275487 15825 leader_election.cc:341] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:24.275607 15825 leader_election.cc:302] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:24.276329 17680 consensus_queue.cc:260] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.277016 17185 raft_consensus.cc:2726] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:24.278976 17680 ts_tablet_manager.cc:1398] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240501 08:12:24.280077 12830 catalog_manager.cc:5484] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:24.283272 17680 tablet_bootstrap.cc:492] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:24.285616 15735 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:24.290410 17680 tablet_bootstrap.cc:654] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.299620 17680 tablet_bootstrap.cc:492] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:24.299839 17680 ts_tablet_manager.cc:1367] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 08:12:24.300591 17679 tablet_bootstrap.cc:654] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:24.300784 17156 raft_consensus.cc:2781] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.300930 17156 raft_consensus.cc:482] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.301067 17156 raft_consensus.cc:3037] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.300972 17680 raft_consensus.cc:348] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.301250 17680 raft_consensus.cc:374] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.301368 17680 raft_consensus.cc:729] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.301599 17680 consensus_queue.cc:260] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.302129 17680 ts_tablet_manager.cc:1398] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:24.303903 17685 tablet_bootstrap.cc:492] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:24.303784 17156 raft_consensus.cc:504] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.304097 17685 ts_tablet_manager.cc:1367] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.171s user 0.003s sys 0.011s
I20240501 08:12:24.306011 17685 raft_consensus.cc:348] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.306330 17685 raft_consensus.cc:374] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.306478 17685 raft_consensus.cc:729] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.306774 17685 consensus_queue.cc:260] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.312489 13231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279b5f4d8fba4c12beec040c6ed39992" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:24.312791 13231 raft_consensus.cc:3037] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.312968 17685 ts_tablet_manager.cc:1398] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:24.315996 13231 raft_consensus.cc:2454] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:24.318492 15960 leader_election.cc:302] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:24.318876 17653 raft_consensus.cc:2781] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.319517 17156 leader_election.cc:288] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:24.319922 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279b5f4d8fba4c12beec040c6ed39992" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:24.320145 15893 raft_consensus.cc:3037] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.323444 15893 raft_consensus.cc:2454] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:24.333856 17679 tablet_bootstrap.cc:492] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:24.334101 17679 ts_tablet_manager.cc:1367] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.001s
I20240501 08:12:24.334754 17679 raft_consensus.cc:348] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.334964 17679 raft_consensus.cc:374] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:24.335037 17679 raft_consensus.cc:729] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:24.335290 17679 consensus_queue.cc:260] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.335781 17679 ts_tablet_manager.cc:1398] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:24.344017 17653 raft_consensus.cc:686] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:24.345366 17653 consensus_queue.cc:237] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.358278 12826 catalog_manager.cc:5484] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:24.367300 17185 raft_consensus.cc:482] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.367566 17185 raft_consensus.cc:504] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.368439 17185 leader_election.cc:288] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:24.369338 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a8174993804850839b063726d66df5" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:24.369680 15754 raft_consensus.cc:2454] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:24.370551 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a8174993804850839b063726d66df5" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:24.370820 16154 raft_consensus.cc:2454] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:24.371467 15825 leader_election.cc:302] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:24.373474 17185 raft_consensus.cc:2781] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.373656 17185 raft_consensus.cc:482] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.373766 17185 raft_consensus.cc:3037] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.377676 17185 raft_consensus.cc:504] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.378995 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a8174993804850839b063726d66df5" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:24.379241 16154 raft_consensus.cc:3037] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.379920 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a8174993804850839b063726d66df5" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:24.380156 15754 raft_consensus.cc:3037] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.383085 16154 raft_consensus.cc:2454] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:24.383641 17185 leader_election.cc:288] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:24.384011 15823 leader_election.cc:302] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:24.384452 17472 raft_consensus.cc:2781] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.384665 17472 raft_consensus.cc:686] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:24.384824 15754 raft_consensus.cc:2454] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:24.384970 17472 consensus_queue.cc:237] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:24.389281 12833 catalog_manager.cc:5484] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:24.410377 17588 raft_consensus.cc:482] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.410588 17588 raft_consensus.cc:504] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:24.411497 17588 leader_election.cc:288] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:24.411959 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714a84eae9324caeb4f25e798d6c35c2" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:24.412055 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714a84eae9324caeb4f25e798d6c35c2" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:24.412439 15754 raft_consensus.cc:2379] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cc32f6e2d49477f9a23ce6e444039fb in current term 1: Already voted for candidate 76d830e199eb461c8ca689c9460ee0c2 in this term.
I20240501 08:12:24.412961 16092 leader_election.cc:302] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:24.413525 17588 raft_consensus.cc:3037] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.417160 17588 raft_consensus.cc:2726] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:24.435981 13231 consensus_queue.cc:237] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:24.438226 16024 raft_consensus.cc:1261] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:24.438289 13103 raft_consensus.cc:1261] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:24.438866 17218 consensus_queue.cc:1035] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:24.439198 17399 consensus_queue.cc:1035] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:24.441949 17399 raft_consensus.cc:2932] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:24.442704 17588 raft_consensus.cc:482] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:24.442893 17588 raft_consensus.cc:504] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.443657 17588 leader_election.cc:288] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:24.443512 13103 raft_consensus.cc:2932] T 94e7fb526fbc4ebc94875c00b49ac43d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:24.443588 16024 raft_consensus.cc:2932] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:24.446352 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9259925ce64e4b93afc8b08045c19cd9" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:24.446599 15893 raft_consensus.cc:2454] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:24.447118 16090 leader_election.cc:302] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:24.447425 17588 raft_consensus.cc:2781] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:24.447600 17588 raft_consensus.cc:482] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:24.447690 17588 raft_consensus.cc:3037] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.448165 12833 catalog_manager.cc:5484] T 94e7fb526fbc4ebc94875c00b49ac43d P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:24.449527 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9259925ce64e4b93afc8b08045c19cd9" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:24.449767 16025 raft_consensus.cc:2454] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:24.450629 17588 raft_consensus.cc:504] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.451773 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9259925ce64e4b93afc8b08045c19cd9" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:24.452055 15893 raft_consensus.cc:3037] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.454839 15893 raft_consensus.cc:2454] T 9259925ce64e4b93afc8b08045c19cd9 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:24.455322 16090 leader_election.cc:302] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:24.455649 17690 raft_consensus.cc:2781] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:24.457844 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e7fb526fbc4ebc94875c00b49ac43d with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:24.458990 17588 leader_election.cc:288] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:24.459028 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9259925ce64e4b93afc8b08045c19cd9" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:24.459342 16025 raft_consensus.cc:3037] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:24.463112 16025 raft_consensus.cc:2454] T 9259925ce64e4b93afc8b08045c19cd9 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:24.469714 17731 tablet_replica.cc:330] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:24.470081 17731 raft_consensus.cc:2229] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:24.473021 17731 raft_consensus.cc:2258] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:24.474011 17731 ts_tablet_manager.cc:1875] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:24.479665 17731 ts_tablet_manager.cc:1888] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:24.479854 17731 log.cc:1196] T 94e7fb526fbc4ebc94875c00b49ac43d P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/94e7fb526fbc4ebc94875c00b49ac43d
I20240501 08:12:24.480846 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 94e7fb526fbc4ebc94875c00b49ac43d (table rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]) successfully deleted
I20240501 08:12:24.484562 17690 raft_consensus.cc:686] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:24.484933 17690 consensus_queue.cc:237] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:24.489720 12826 catalog_manager.cc:5484] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:24.506295 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:24.524240 15754 raft_consensus.cc:1261] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:24.524451 16154 raft_consensus.cc:1261] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:24.525074 17472 consensus_queue.cc:1035] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.525399 17185 consensus_queue.cc:1035] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.552330 12826 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 53a8174993804850839b063726d66df5 (table rebalancer_dynamic_table [id=1442e0ca3ea042b5a8980acede0c0735]). Expected version 1 got 0
I20240501 08:12:24.570431 17746 tablet.cc:1722] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:24.571559 12833 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet 53a8174993804850839b063726d66df5 (table rebalancer_dynamic_table [id=1442e0ca3ea042b5a8980acede0c0735]). Expected version 1 got 0
I20240501 08:12:24.574070 12833 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 53a8174993804850839b063726d66df5 (table rebalancer_dynamic_table [id=1442e0ca3ea042b5a8980acede0c0735]). Expected version 1 got 0
I20240501 08:12:24.574292 17746 tablet.cc:1620] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:24.582201 17745 tablet.cc:1722] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:24.583369 17745 tablet.cc:1620] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:24.585047 17750 tablet.cc:1722] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:24.588966 17750 tablet.cc:1620] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:24.591710 12826 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet 53a8174993804850839b063726d66df5 (table rebalancer_dynamic_table [id=1442e0ca3ea042b5a8980acede0c0735]). Expected version 1 got 0
I20240501 08:12:24.596261 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=1442e0ca3ea042b5a8980acede0c0735] alter complete (version 1)
I20240501 08:12:24.705744 12825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:24.719949 17745 tablet.cc:1722] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:24.720541 17745 tablet.cc:1620] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:24.722002 17746 tablet.cc:1722] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:24.722512 17746 tablet.cc:1620] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:24.725497 17750 tablet.cc:1722] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:24.726052 17750 tablet.cc:1620] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:24.740669 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=1442e0ca3ea042b5a8980acede0c0735] alter complete (version 2)
I20240501 08:12:24.809618 13345 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:24.818058 17653 consensus_queue.cc:1035] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.823266 13214 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:24.835229 16005 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:24.845237 12827 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:24.845427 17678 consensus_queue.cc:1035] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.849362 17701 consensus_queue.cc:1035] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.854745 17745 tablet.cc:1722] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:24.857889 17653 consensus_queue.cc:1035] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:24.864266 17750 tablet.cc:1722] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:24.866271 17746 tablet.cc:1722] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:24.876965 12953 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:24.922094 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=1442e0ca3ea042b5a8980acede0c0735] alter complete (version 3)
I20240501 08:12:24.924495 16138 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:12:24.952782 15735 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:24.956513 15871 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:24.962718 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:25.010471 17690 consensus_queue.cc:1035] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.023221 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:25.028949 12827 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:25.029726 12827 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:25.035084 12827 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 53a8174993804850839b063726d66df5
I20240501 08:12:25.036265 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8174993804850839b063726d66df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:25.036505 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8174993804850839b063726d66df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:25.037067 15871 tablet_service.cc:1514] Processing DeleteTablet for tablet 53a8174993804850839b063726d66df5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:25.041824 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 6840d246ddfa42b3b8065d7136a36f5a (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.042349 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6840d246ddfa42b3b8065d7136a36f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.042362 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 6840d246ddfa42b3b8065d7136a36f5a (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.042750 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6840d246ddfa42b3b8065d7136a36f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.042785 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 34a5baf3631b45d68a2003fc4539f799 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.043221 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a5baf3631b45d68a2003fc4539f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.044664 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 6840d246ddfa42b3b8065d7136a36f5a (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.045094 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6840d246ddfa42b3b8065d7136a36f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.045682 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 34a5baf3631b45d68a2003fc4539f799 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.046180 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a5baf3631b45d68a2003fc4539f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.049788 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 34a5baf3631b45d68a2003fc4539f799 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.050212 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a5baf3631b45d68a2003fc4539f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.053936 15733 tablet_service.cc:1467] Processing CreateTablet for tablet cabfd31e8ee941cf9f69bc04a0510d11 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.054392 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabfd31e8ee941cf9f69bc04a0510d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.054765 16137 tablet_service.cc:1467] Processing CreateTablet for tablet cabfd31e8ee941cf9f69bc04a0510d11 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.054914 15870 tablet_service.cc:1467] Processing CreateTablet for tablet cabfd31e8ee941cf9f69bc04a0510d11 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:25.055194 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabfd31e8ee941cf9f69bc04a0510d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.055419 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabfd31e8ee941cf9f69bc04a0510d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.060425 17690 consensus_queue.cc:1035] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.090476 17815 tablet_bootstrap.cc:492] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:25.092092 17814 tablet_bootstrap.cc:492] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:25.094713 17814 tablet_bootstrap.cc:654] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.096508 17815 tablet_bootstrap.cc:654] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.100489 17812 tablet_bootstrap.cc:492] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:25.104277 17812 tablet_bootstrap.cc:654] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.107229 17813 tablet_bootstrap.cc:492] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:25.110219 17813 tablet_bootstrap.cc:654] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.110924 17817 tablet_bootstrap.cc:492] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:25.114522 17817 tablet_bootstrap.cc:654] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.127336 17809 tablet_replica.cc:330] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:25.127789 17809 raft_consensus.cc:2229] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:25.128351 17809 raft_consensus.cc:2258] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.129371 17809 ts_tablet_manager.cc:1875] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:25.103948 17808 tablet_replica.cc:330] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:25.132591 17808 raft_consensus.cc:2229] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:25.132874 17808 raft_consensus.cc:2258] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.133983 17808 ts_tablet_manager.cc:1875] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:25.137244 17813 tablet_bootstrap.cc:492] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:25.137398 17813 ts_tablet_manager.cc:1367] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240501 08:12:25.138049 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:25.140827 17817 tablet_bootstrap.cc:492] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:25.140132 17812 tablet_bootstrap.cc:492] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:25.141155 17812 ts_tablet_manager.cc:1367] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20240501 08:12:25.144269 17807 tablet_replica.cc:330] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:25.144580 17807 raft_consensus.cc:2229] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:25.144901 17807 raft_consensus.cc:2258] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.146296 17807 ts_tablet_manager.cc:1875] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:25.146808 17817 ts_tablet_manager.cc:1367] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.001s
I20240501 08:12:25.148247 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59aff02f252c4798af9d555d78558944]), partition=
I20240501 08:12:25.148626 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.149549 15871 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59aff02f252c4798af9d555d78558944]), partition=
I20240501 08:12:25.149871 15871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.152773 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=59aff02f252c4798af9d555d78558944]), partition=
I20240501 08:12:25.153218 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:25.163156 17812 raft_consensus.cc:348] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:25.163522 17812 raft_consensus.cc:374] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.163664 17812 raft_consensus.cc:729] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.163975 17812 consensus_queue.cc:260] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:25.164507 17817 raft_consensus.cc:348] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.164678 17808 ts_tablet_manager.cc:1888] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:25.169294 17808 log.cc:1196] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/53a8174993804850839b063726d66df5
I20240501 08:12:25.168113 17814 tablet_bootstrap.cc:492] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:25.169481 17817 raft_consensus.cc:374] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.169673 17814 ts_tablet_manager.cc:1367] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.000s
I20240501 08:12:25.138193 17813 raft_consensus.cc:348] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.170046 17817 raft_consensus.cc:729] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.175963 17814 raft_consensus.cc:348] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.176173 17814 raft_consensus.cc:374] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.176251 17814 raft_consensus.cc:729] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.176615 17814 consensus_queue.cc:260] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.177556 17814 ts_tablet_manager.cc:1398] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:12:25.177898 17814 tablet_bootstrap.cc:492] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:25.179558 17814 tablet_bootstrap.cc:654] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.179670 17808 ts_tablet_manager.cc:1909] T 53a8174993804850839b063726d66df5 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:25.181303 17813 raft_consensus.cc:374] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.181720 17813 raft_consensus.cc:729] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.182053 17817 consensus_queue.cc:260] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.183133 17812 ts_tablet_manager.cc:1398] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240501 08:12:25.184015 17815 tablet_bootstrap.cc:492] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:25.184176 17815 ts_tablet_manager.cc:1367] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.094s user 0.004s sys 0.016s
I20240501 08:12:25.184445 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 53a8174993804850839b063726d66df5 (table rebalancer_dynamic_table_new_name [id=1442e0ca3ea042b5a8980acede0c0735]) successfully deleted
I20240501 08:12:25.185645 17812 tablet_bootstrap.cc:492] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:25.187178 17815 raft_consensus.cc:348] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.187394 17815 raft_consensus.cc:374] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.182929 17813 consensus_queue.cc:260] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.187490 17815 raft_consensus.cc:729] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.187716 17815 consensus_queue.cc:260] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.188004 17813 ts_tablet_manager.cc:1398] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240501 08:12:25.188620 17815 ts_tablet_manager.cc:1398] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:25.188901 17815 tablet_bootstrap.cc:492] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:25.190699 17823 tablet_bootstrap.cc:492] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:25.187256 17817 ts_tablet_manager.cc:1398] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.040s user 0.000s sys 0.002s
I20240501 08:12:25.195869 17817 tablet_bootstrap.cc:492] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:25.197547 17812 tablet_bootstrap.cc:654] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.199277 17823 tablet_bootstrap.cc:654] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.200268 17817 tablet_bootstrap.cc:654] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.206081 17812 tablet_bootstrap.cc:492] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:25.206279 17812 ts_tablet_manager.cc:1367] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240501 08:12:25.208454 17812 raft_consensus.cc:348] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.211297 17812 raft_consensus.cc:374] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.211396 17812 raft_consensus.cc:729] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.211655 17812 consensus_queue.cc:260] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.212400 17812 ts_tablet_manager.cc:1398] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240501 08:12:25.213094 17815 tablet_bootstrap.cc:654] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.214401 17817 tablet_bootstrap.cc:492] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:25.214978 17817 ts_tablet_manager.cc:1367] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240501 08:12:25.216943 17817 raft_consensus.cc:348] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.217255 17817 raft_consensus.cc:374] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.217367 17817 raft_consensus.cc:729] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.217679 17817 consensus_queue.cc:260] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.218331 17817 ts_tablet_manager.cc:1398] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:25.220039 17809 ts_tablet_manager.cc:1888] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:25.220968 17809 log.cc:1196] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/53a8174993804850839b063726d66df5
I20240501 08:12:25.223202 17809 ts_tablet_manager.cc:1909] T 53a8174993804850839b063726d66df5 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:25.230058 17805 raft_consensus.cc:482] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:25.230227 17805 raft_consensus.cc:504] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.230962 17805 leader_election.cc:288] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:25.231840 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabfd31e8ee941cf9f69bc04a0510d11" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
W20240501 08:12:25.232661 16092 leader_election.cc:341] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:25.233453 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabfd31e8ee941cf9f69bc04a0510d11" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:25.233718 15892 raft_consensus.cc:2454] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:25.234678 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 53a8174993804850839b063726d66df5 (table rebalancer_dynamic_table_new_name [id=1442e0ca3ea042b5a8980acede0c0735]) successfully deleted
I20240501 08:12:25.235210 16090 leader_election.cc:302] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:25.235613 17805 raft_consensus.cc:2781] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:25.235711 17805 raft_consensus.cc:482] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:25.235785 17805 raft_consensus.cc:3037] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.238715 17805 raft_consensus.cc:504] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.240694 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabfd31e8ee941cf9f69bc04a0510d11" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
W20240501 08:12:25.241473 16092 leader_election.cc:341] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:25.242945 17815 tablet_bootstrap.cc:492] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:25.243197 17815 ts_tablet_manager.cc:1367] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.010s
I20240501 08:12:25.244251 17815 raft_consensus.cc:348] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:25.244504 17815 raft_consensus.cc:374] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.244575 17815 raft_consensus.cc:729] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.244839 17815 consensus_queue.cc:260] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:25.246672 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabfd31e8ee941cf9f69bc04a0510d11" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:25.246860 15893 raft_consensus.cc:3037] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.249642 17815 ts_tablet_manager.cc:1398] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:25.250015 17815 tablet_bootstrap.cc:492] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:25.249616 15893 raft_consensus.cc:2454] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:25.252966 16090 leader_election.cc:302] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:25.253332 17805 leader_election.cc:288] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:25.253569 17690 raft_consensus.cc:2781] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:25.253780 17690 raft_consensus.cc:686] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:25.254127 17690 consensus_queue.cc:237] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.254377 17823 tablet_bootstrap.cc:492] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:25.254540 17823 ts_tablet_manager.cc:1367] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.013s
I20240501 08:12:25.255352 17823 raft_consensus.cc:348] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.255597 17823 raft_consensus.cc:374] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.255733 17823 raft_consensus.cc:729] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.256072 17823 consensus_queue.cc:260] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.258846 12826 catalog_manager.cc:5484] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:25.262214 17823 ts_tablet_manager.cc:1398] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20240501 08:12:25.266929 17807 ts_tablet_manager.cc:1888] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:25.267120 17807 log.cc:1196] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/53a8174993804850839b063726d66df5
I20240501 08:12:25.267541 17815 tablet_bootstrap.cc:654] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.267752 17807 ts_tablet_manager.cc:1909] T 53a8174993804850839b063726d66df5 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:25.276180 17815 tablet_bootstrap.cc:492] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:25.276386 17815 ts_tablet_manager.cc:1367] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.007s
I20240501 08:12:25.277165 17815 raft_consensus.cc:348] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.277374 17815 raft_consensus.cc:374] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.277470 17815 raft_consensus.cc:729] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.277724 17815 consensus_queue.cc:260] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.280689 17815 ts_tablet_manager.cc:1398] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:25.284771 17810 raft_consensus.cc:482] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:25.284991 17810 raft_consensus.cc:504] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.286041 17810 leader_election.cc:288] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:25.287806 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e62a7d18d4804b3a0106d1f3f0f20" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:25.288080 15893 raft_consensus.cc:2454] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:25.289526 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e62a7d18d4804b3a0106d1f3f0f20" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:25.289774 16025 raft_consensus.cc:2454] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:25.290670 13040 leader_election.cc:302] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:25.291105 17810 raft_consensus.cc:2781] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:25.291210 17810 raft_consensus.cc:482] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:25.291290 17810 raft_consensus.cc:3037] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.293831 17810 raft_consensus.cc:504] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.296772 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e62a7d18d4804b3a0106d1f3f0f20" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:25.297025 16025 raft_consensus.cc:3037] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.299085 16025 raft_consensus.cc:2454] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:25.299631 13040 leader_election.cc:302] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:25.300452 17832 raft_consensus.cc:2781] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:25.300782 17832 raft_consensus.cc:686] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:25.300937 17810 leader_election.cc:288] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:25.301318 17832 consensus_queue.cc:237] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.301595 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c3e62a7d18d4804b3a0106d1f3f0f20" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:25.301906 15893 raft_consensus.cc:3037] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.309206 12826 catalog_manager.cc:5484] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:25.311722 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 53a8174993804850839b063726d66df5 (table rebalancer_dynamic_table_new_name [id=1442e0ca3ea042b5a8980acede0c0735]) successfully deleted
I20240501 08:12:25.314239 17814 tablet_bootstrap.cc:492] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:25.314503 17814 ts_tablet_manager.cc:1367] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.137s user 0.030s sys 0.058s
I20240501 08:12:25.315368 17814 raft_consensus.cc:348] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:25.315642 17814 raft_consensus.cc:374] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.315766 17814 raft_consensus.cc:729] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.316102 17814 consensus_queue.cc:260] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:25.316830 17814 ts_tablet_manager.cc:1398] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:25.317159 17814 tablet_bootstrap.cc:492] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:25.319242 17814 tablet_bootstrap.cc:654] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:25.320353 15893 raft_consensus.cc:2454] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:25.340385 17814 tablet_bootstrap.cc:492] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:25.340639 17814 ts_tablet_manager.cc:1367] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.011s
I20240501 08:12:25.341529 17814 raft_consensus.cc:348] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.341810 17814 raft_consensus.cc:374] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:25.341943 17814 raft_consensus.cc:729] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:25.342278 17814 consensus_queue.cc:260] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.343019 17814 ts_tablet_manager.cc:1398] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:25.369611 17678 raft_consensus.cc:482] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:25.369778 17678 raft_consensus.cc:504] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:25.370476 17678 leader_election.cc:288] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:25.370981 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a5baf3631b45d68a2003fc4539f799" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:25.371222 15893 raft_consensus.cc:2454] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:25.371553 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a5baf3631b45d68a2003fc4539f799" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:25.371824 15755 raft_consensus.cc:2454] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:25.372457 12909 leader_election.cc:302] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:25.372754 17678 raft_consensus.cc:2781] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:25.372910 17678 raft_consensus.cc:482] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:25.373005 17678 raft_consensus.cc:3037] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.375353 17678 raft_consensus.cc:504] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:25.376047 17678 leader_election.cc:288] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:25.376487 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a5baf3631b45d68a2003fc4539f799" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:25.376735 15893 raft_consensus.cc:3037] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.377096 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a5baf3631b45d68a2003fc4539f799" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:25.377506 15755 raft_consensus.cc:3037] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.379551 15893 raft_consensus.cc:2454] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:25.380182 15755 raft_consensus.cc:2454] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:25.380204 12907 leader_election.cc:302] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:25.380659 17678 raft_consensus.cc:2781] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:25.380904 17678 raft_consensus.cc:686] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:25.381165 17678 consensus_queue.cc:237] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:25.385412 12827 catalog_manager.cc:5484] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:25.392532 12973 consensus_queue.cc:237] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:25.394507 15755 raft_consensus.cc:1261] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:25.394614 13363 raft_consensus.cc:1261] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:25.395152 17701 consensus_queue.cc:1035] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:25.396157 17701 consensus_queue.cc:1035] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:25.398628 17678 raft_consensus.cc:2932] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:25.399292 13363 raft_consensus.cc:2932] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:25.404181 12827 catalog_manager.cc:5484] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:25.408349 13214 tablet_service.cc:1514] Processing DeleteTablet for tablet cd9a5b5333d54f6d9d8186e4d9b1b1be with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:25.409511 17858 tablet_replica.cc:330] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:25.409897 17858 raft_consensus.cc:2229] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:25.410207 17858 raft_consensus.cc:2258] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:25.411005 17858 ts_tablet_manager.cc:1875] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:25.418043 17858 ts_tablet_manager.cc:1888] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:25.418329 17858 log.cc:1196] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/cd9a5b5333d54f6d9d8186e4d9b1b1be
I20240501 08:12:25.420341 15754 raft_consensus.cc:2932] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:25.421134 12832 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:25.423110 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet cd9a5b5333d54f6d9d8186e4d9b1b1be (table rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]) successfully deleted
I20240501 08:12:25.428159 15893 raft_consensus.cc:1261] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:25.428732 17318 consensus_queue.cc:1035] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.428891 16025 raft_consensus.cc:1261] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:25.429533 17318 consensus_queue.cc:1035] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.435366 12827 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (table rebalancer_dynamic_table [id=59aff02f252c4798af9d555d78558944]). Expected version 1 got 0
I20240501 08:12:25.436542 17652 raft_consensus.cc:482] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:25.436758 17652 raft_consensus.cc:504] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.437602 17652 leader_election.cc:288] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:25.437935 12832 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (table rebalancer_dynamic_table [id=59aff02f252c4798af9d555d78558944]). Expected version 1 got 0
I20240501 08:12:25.438284 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6840d246ddfa42b3b8065d7136a36f5a" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:25.438544 12973 raft_consensus.cc:2454] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:25.439407 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6840d246ddfa42b3b8065d7136a36f5a" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:25.439642 13103 raft_consensus.cc:2454] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:25.440186 15692 leader_election.cc:302] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:25.441012 17652 raft_consensus.cc:2781] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:25.441110 17652 raft_consensus.cc:482] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:25.441226 17652 raft_consensus.cc:3037] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.447095 17652 raft_consensus.cc:504] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.448633 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6840d246ddfa42b3b8065d7136a36f5a" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:25.448947 12973 raft_consensus.cc:3037] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.452919 12973 raft_consensus.cc:2454] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:25.453481 15692 leader_election.cc:302] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:25.453812 17831 raft_consensus.cc:2781] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:25.460048 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6840d246ddfa42b3b8065d7136a36f5a" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:25.460348 13103 raft_consensus.cc:3037] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.463624 13103 raft_consensus.cc:2454] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:25.471053 17652 leader_election.cc:288] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:25.476341 17873 tablet.cc:1722] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:25.481930 17872 tablet.cc:1722] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:25.482532 17872 tablet.cc:1620] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:25.483327 12827 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (table rebalancer_dynamic_table [id=59aff02f252c4798af9d555d78558944]). Expected version 1 got 0
I20240501 08:12:25.485836 17873 tablet.cc:1620] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:25.496745 12827 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (table rebalancer_dynamic_table [id=59aff02f252c4798af9d555d78558944]). Expected version 1 got 0
I20240501 08:12:25.493455 17874 tablet.cc:1722] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:25.504321 17874 tablet.cc:1620] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:25.504740 17831 raft_consensus.cc:686] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:25.505137 17831 consensus_queue.cc:237] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:25.511488 12827 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=59aff02f252c4798af9d555d78558944] alter complete (version 1)
I20240501 08:12:25.520609 12827 catalog_manager.cc:5484] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:25.622332 12832 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:25.630570 17872 tablet.cc:1722] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:25.631071 17872 tablet.cc:1620] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:25.631454 17874 tablet.cc:1722] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:25.631520 17873 tablet.cc:1722] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:25.631997 17874 tablet.cc:1620] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:25.632021 17873 tablet.cc:1620] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:25.649173 12827 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=59aff02f252c4798af9d555d78558944] alter complete (version 2)
I20240501 08:12:25.744453 15871 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:25.762214 13345 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:25.767623 12833 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:25.769742 17831 raft_consensus.cc:482] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:25.769922 17831 raft_consensus.cc:504] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:25.770629 17831 leader_election.cc:288] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:25.771521 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabfd31e8ee941cf9f69bc04a0510d11" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:25.775014 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cabfd31e8ee941cf9f69bc04a0510d11" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:25.775316 15892 raft_consensus.cc:2379] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 982f49fc87f149439cda0de2261b6bfd in current term 1: Already voted for candidate 6cc32f6e2d49477f9a23ce6e444039fb in this term.
I20240501 08:12:25.777058 15689 leader_election.cc:302] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:25.777138 13214 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:25.777472 17831 raft_consensus.cc:3037] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:25.779151 17690 consensus_queue.cc:1035] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.782929 17831 raft_consensus.cc:2726] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:25.790999 15735 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:25.814811 16005 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:25.836330 17872 tablet.cc:1722] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:25.838241 17874 tablet.cc:1722] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:25.841274 12954 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:25.849464 17873 tablet.cc:1722] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:25.849829 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=59aff02f252c4798af9d555d78558944] alter complete (version 3)
I20240501 08:12:25.897691 13084 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:25.902696 17831 consensus_queue.cc:1035] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.945987 16138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240501 08:12:25.948064 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.69:42897, user_credentials={real_user=slave}} blocked reactor thread for 74386.7us
I20240501 08:12:25.949884 17925 consensus_queue.cc:1035] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.960309 17678 consensus_queue.cc:1035] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.965591 17678 consensus_queue.cc:1035] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:25.967936 12833 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:25.968199 12833 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:25.972527 12833 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 6c3e62a7d18d4804b3a0106d1f3f0f20
I20240501 08:12:25.974628 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:25.975386 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:25.976305 15871 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:25 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:25.982317 17875 consensus_queue.cc:1035] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.000425 17935 tablet_replica.cc:330] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:26.000802 17935 raft_consensus.cc:2229] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:26.001562 17935 raft_consensus.cc:2258] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:26.003360 17935 ts_tablet_manager.cc:1875] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:26.030028 17934 tablet_replica.cc:330] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:26.030279 17934 raft_consensus.cc:2229] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:26.030658 17934 raft_consensus.cc:2258] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:26.031265 17936 tablet_replica.cc:330] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:26.031596 17936 raft_consensus.cc:2229] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:26.031894 17936 raft_consensus.cc:2258] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:26.033855 17934 ts_tablet_manager.cc:1875] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:26.041530 17936 ts_tablet_manager.cc:1875] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:26.044093 12826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:26.054879 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 608bc35666234d2cbcb816439d050196 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.055117 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 608bc35666234d2cbcb816439d050196 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.055338 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608bc35666234d2cbcb816439d050196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.055562 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608bc35666234d2cbcb816439d050196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.055958 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 6aa998326f3c4ced9afbee7e54bffa0e (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.056406 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa998326f3c4ced9afbee7e54bffa0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.056663 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 5e61276f6f4c40f8987747b997ac52ac (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.057046 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e61276f6f4c40f8987747b997ac52ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.060523 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 5e61276f6f4c40f8987747b997ac52ac (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.060931 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e61276f6f4c40f8987747b997ac52ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.060930 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 608bc35666234d2cbcb816439d050196 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.061370 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608bc35666234d2cbcb816439d050196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.061641 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 5e61276f6f4c40f8987747b997ac52ac (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.061817 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 6aa998326f3c4ced9afbee7e54bffa0e (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.062016 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e61276f6f4c40f8987747b997ac52ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.062239 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa998326f3c4ced9afbee7e54bffa0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.065944 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 6aa998326f3c4ced9afbee7e54bffa0e (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.066336 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa998326f3c4ced9afbee7e54bffa0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.075783 12826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:26.085335 16004 tablet_service.cc:1467] Processing CreateTablet for tablet c8f25679cb4c4d69aa2864726088ed3e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f3a45aa4df9948c3881fd116481daefd]), partition=
I20240501 08:12:26.085690 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f25679cb4c4d69aa2864726088ed3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.087726 15733 tablet_service.cc:1467] Processing CreateTablet for tablet c8f25679cb4c4d69aa2864726088ed3e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f3a45aa4df9948c3881fd116481daefd]), partition=
I20240501 08:12:26.088163 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f25679cb4c4d69aa2864726088ed3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.095731 15872 tablet_service.cc:1467] Processing CreateTablet for tablet c8f25679cb4c4d69aa2864726088ed3e (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f3a45aa4df9948c3881fd116481daefd]), partition=
I20240501 08:12:26.096171 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f25679cb4c4d69aa2864726088ed3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.099023 17943 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:26.099257 17940 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:26.103507 17943 tablet_bootstrap.cc:654] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.104545 17935 ts_tablet_manager.cc:1888] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:26.104753 17935 log.cc:1196] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/6c3e62a7d18d4804b3a0106d1f3f0f20
I20240501 08:12:26.105355 17935 ts_tablet_manager.cc:1909] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:26.108822 17940 tablet_bootstrap.cc:654] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.110121 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (table rebalancer_dynamic_table_new_name [id=59aff02f252c4798af9d555d78558944]) successfully deleted
I20240501 08:12:26.121945 17934 ts_tablet_manager.cc:1888] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:26.127022 17941 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:26.129317 17939 tablet_bootstrap.cc:492] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:26.131314 17934 log.cc:1196] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/6c3e62a7d18d4804b3a0106d1f3f0f20
I20240501 08:12:26.131441 17941 tablet_bootstrap.cc:654] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.132485 17943 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:26.132701 17943 ts_tablet_manager.cc:1367] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240501 08:12:26.133246 17934 ts_tablet_manager.cc:1909] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:26.133651 17943 raft_consensus.cc:348] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.133888 17943 raft_consensus.cc:374] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.134012 17943 raft_consensus.cc:729] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.134269 17943 consensus_queue.cc:260] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.135365 17940 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:26.135561 17940 ts_tablet_manager.cc:1367] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20240501 08:12:26.136344 17943 ts_tablet_manager.cc:1398] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:26.136868 17943 tablet_bootstrap.cc:492] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:26.138607 17943 tablet_bootstrap.cc:654] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.142660 17940 raft_consensus.cc:348] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.142964 17940 raft_consensus.cc:374] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.143118 17940 raft_consensus.cc:729] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.143471 17940 consensus_queue.cc:260] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.147388 17940 ts_tablet_manager.cc:1398] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240501 08:12:26.151110 17940 tablet_bootstrap.cc:492] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:26.152400 17936 ts_tablet_manager.cc:1888] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:26.153000 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (table rebalancer_dynamic_table_new_name [id=59aff02f252c4798af9d555d78558944]) successfully deleted
I20240501 08:12:26.153565 17936 log.cc:1196] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/6c3e62a7d18d4804b3a0106d1f3f0f20
I20240501 08:12:26.154110 17936 ts_tablet_manager.cc:1909] T 6c3e62a7d18d4804b3a0106d1f3f0f20 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:26.155161 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 6c3e62a7d18d4804b3a0106d1f3f0f20 (table rebalancer_dynamic_table_new_name [id=59aff02f252c4798af9d555d78558944]) successfully deleted
I20240501 08:12:26.153457 17939 tablet_bootstrap.cc:654] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.155853 17941 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:26.156015 17941 ts_tablet_manager.cc:1367] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240501 08:12:26.156980 17941 raft_consensus.cc:348] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.157366 17941 raft_consensus.cc:374] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.157466 17941 raft_consensus.cc:729] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.157716 17941 consensus_queue.cc:260] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.158855 17941 ts_tablet_manager.cc:1398] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:26.159266 17941 tablet_bootstrap.cc:492] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:26.161564 17941 tablet_bootstrap.cc:654] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.162608 17940 tablet_bootstrap.cc:654] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.162884 17943 tablet_bootstrap.cc:492] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:26.163121 17943 ts_tablet_manager.cc:1367] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.004s
I20240501 08:12:26.164844 17943 raft_consensus.cc:348] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:26.166981 17943 raft_consensus.cc:374] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.167114 17943 raft_consensus.cc:729] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.167467 17943 consensus_queue.cc:260] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:26.169284 17943 ts_tablet_manager.cc:1398] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:26.169638 17943 tablet_bootstrap.cc:492] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:26.173489 17940 tablet_bootstrap.cc:492] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:26.173722 17940 ts_tablet_manager.cc:1367] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20240501 08:12:26.174535 17940 raft_consensus.cc:348] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:26.174825 17940 raft_consensus.cc:374] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.174961 17940 raft_consensus.cc:729] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.175299 17940 consensus_queue.cc:260] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:26.179450 17940 ts_tablet_manager.cc:1398] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 08:12:26.180521 17831 raft_consensus.cc:482] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.180768 17943 tablet_bootstrap.cc:654] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.180703 17831 raft_consensus.cc:504] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.181465 17831 leader_election.cc:288] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:26.182219 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:26.182497 16025 raft_consensus.cc:2454] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:26.184520 15691 leader_election.cc:302] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:26.185613 17831 raft_consensus.cc:2781] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.185767 17831 raft_consensus.cc:482] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.185941 17831 raft_consensus.cc:3037] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.187878 17940 tablet_bootstrap.cc:492] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:26.191932 17831 raft_consensus.cc:504] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.192838 17943 tablet_bootstrap.cc:492] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:26.192984 17943 ts_tablet_manager.cc:1367] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.002s
I20240501 08:12:26.193652 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:26.193785 17943 raft_consensus.cc:348] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:26.193965 17943 raft_consensus.cc:374] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.193948 16025 raft_consensus.cc:3037] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.194049 17943 raft_consensus.cc:729] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.194366 17943 consensus_queue.cc:260] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:26.195959 17943 ts_tablet_manager.cc:1398] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:12:26.198660 16025 raft_consensus.cc:2454] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:26.199316 15691 leader_election.cc:302] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:26.199741 17875 raft_consensus.cc:2781] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.199939 17875 raft_consensus.cc:686] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:26.200341 17875 consensus_queue.cc:237] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.209631 17831 leader_election.cc:288] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:26.210361 17940 tablet_bootstrap.cc:654] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.218127 17941 tablet_bootstrap.cc:492] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:26.217826 17939 tablet_bootstrap.cc:492] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:26.218354 17941 ts_tablet_manager.cc:1367] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.059s user 0.030s sys 0.015s
I20240501 08:12:26.218503 17939 ts_tablet_manager.cc:1367] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.010s
I20240501 08:12:26.219233 17941 raft_consensus.cc:348] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:26.219537 17941 raft_consensus.cc:374] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.219624 17941 raft_consensus.cc:729] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.219899 17941 consensus_queue.cc:260] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:26.220521 17941 ts_tablet_manager.cc:1398] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:26.220950 17941 tablet_bootstrap.cc:492] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:26.222517 17941 tablet_bootstrap.cc:654] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.226593 17940 tablet_bootstrap.cc:492] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:26.228065 17940 ts_tablet_manager.cc:1367] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20240501 08:12:26.229923 13231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:26.230217 13231 raft_consensus.cc:2454] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:26.230578 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:26.230760 13235 raft_consensus.cc:3037] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.230629 17939 raft_consensus.cc:348] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:26.230908 17939 raft_consensus.cc:374] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.231006 17939 raft_consensus.cc:729] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.231184 17941 tablet_bootstrap.cc:492] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:26.231324 17941 ts_tablet_manager.cc:1367] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240501 08:12:26.231278 17939 consensus_queue.cc:260] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:26.232237 17941 raft_consensus.cc:348] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.232470 17941 raft_consensus.cc:374] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.232566 17941 raft_consensus.cc:729] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.232885 17941 consensus_queue.cc:260] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.233583 17941 ts_tablet_manager.cc:1398] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:26.234561 12833 catalog_manager.cc:5484] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:26.235378 13235 raft_consensus.cc:2454] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:26.236784 17940 raft_consensus.cc:348] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.237008 17940 raft_consensus.cc:374] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.237113 17940 raft_consensus.cc:729] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.237367 17939 ts_tablet_manager.cc:1398] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240501 08:12:26.237351 17940 consensus_queue.cc:260] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.237789 17939 tablet_bootstrap.cc:492] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:26.237919 17940 ts_tablet_manager.cc:1398] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.010s user 0.000s sys 0.000s
I20240501 08:12:26.239549 17939 tablet_bootstrap.cc:654] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.242344 17939 tablet_bootstrap.cc:492] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:26.242609 17939 ts_tablet_manager.cc:1367] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 08:12:26.244082 17939 raft_consensus.cc:348] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:26.244334 17939 raft_consensus.cc:374] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.244407 17939 raft_consensus.cc:729] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.244757 17939 consensus_queue.cc:260] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:26.245378 17939 ts_tablet_manager.cc:1398] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:26.245762 17939 tablet_bootstrap.cc:492] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:26.247262 17939 tablet_bootstrap.cc:654] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.251798 17939 tablet_bootstrap.cc:492] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:26.251983 17939 ts_tablet_manager.cc:1367] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 08:12:26.252744 17939 raft_consensus.cc:348] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.252980 17939 raft_consensus.cc:374] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:26.253084 17939 raft_consensus.cc:729] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:26.253325 17939 consensus_queue.cc:260] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.255959 17939 ts_tablet_manager.cc:1398] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:26.260107 17875 raft_consensus.cc:482] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.260270 17875 raft_consensus.cc:504] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:26.260982 17875 leader_election.cc:288] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:26.262490 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608bc35666234d2cbcb816439d050196" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:26.262743 13235 raft_consensus.cc:2454] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:26.263586 15692 leader_election.cc:302] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:26.264598 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608bc35666234d2cbcb816439d050196" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:26.264850 15892 raft_consensus.cc:2454] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:26.267441 17472 raft_consensus.cc:482] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.267693 17472 raft_consensus.cc:504] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.268584 17472 leader_election.cc:288] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:26.269160 17707 raft_consensus.cc:482] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:26.269341 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f25679cb4c4d69aa2864726088ed3e" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:26.269556 16025 raft_consensus.cc:2454] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:26.269490 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f25679cb4c4d69aa2864726088ed3e" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:26.269685 15754 raft_consensus.cc:2454] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:26.269886 15825 leader_election.cc:302] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:26.269320 17707 raft_consensus.cc:504] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:26.270380 17472 raft_consensus.cc:2781] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.270470 17472 raft_consensus.cc:482] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.270479 17875 raft_consensus.cc:2781] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.270546 17707 leader_election.cc:288] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:26.270578 17875 raft_consensus.cc:482] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.270584 17472 raft_consensus.cc:3037] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.270645 17875 raft_consensus.cc:3037] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.271754 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e61276f6f4c40f8987747b997ac52ac" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:26.271973 15892 raft_consensus.cc:2454] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:26.272480 15957 leader_election.cc:302] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:26.273612 17472 raft_consensus.cc:504] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.273824 17875 raft_consensus.cc:504] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:26.274804 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e61276f6f4c40f8987747b997ac52ac" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:26.275038 13235 raft_consensus.cc:2454] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:26.275177 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f25679cb4c4d69aa2864726088ed3e" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:26.275444 16025 raft_consensus.cc:3037] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.279322 16025 raft_consensus.cc:2454] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:26.280054 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608bc35666234d2cbcb816439d050196" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:26.280686 15892 raft_consensus.cc:3037] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.285274 15825 leader_election.cc:302] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:26.288470 17472 leader_election.cc:288] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:26.290833 17472 raft_consensus.cc:2781] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.291262 17472 raft_consensus.cc:686] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:26.291936 17472 consensus_queue.cc:237] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:26.293982 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "608bc35666234d2cbcb816439d050196" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:26.294256 13235 raft_consensus.cc:3037] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.290465 15892 raft_consensus.cc:2454] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:26.296818 15689 leader_election.cc:302] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:26.297264 17831 raft_consensus.cc:2781] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.297446 17831 raft_consensus.cc:686] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:26.297803 17831 consensus_queue.cc:237] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:26.302042 13235 raft_consensus.cc:2454] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:26.310156 17875 leader_election.cc:288] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:26.312060 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f25679cb4c4d69aa2864726088ed3e" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:26.312376 15754 raft_consensus.cc:3037] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.318434 15754 raft_consensus.cc:2454] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:26.323954 12833 catalog_manager.cc:5484] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:26.329865 17959 raft_consensus.cc:2781] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:26.330004 17959 raft_consensus.cc:482] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:26.330122 17959 raft_consensus.cc:3037] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:12:26.331022 15959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.69:42897, user_credentials={real_user=slave}} blocked reactor thread for 60696.3us
I20240501 08:12:26.332794 12833 catalog_manager.cc:5484] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:26.337150 17959 raft_consensus.cc:504] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:26.337915 17959 leader_election.cc:288] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:26.338315 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e61276f6f4c40f8987747b997ac52ac" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:26.338382 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e61276f6f4c40f8987747b997ac52ac" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:26.338544 13235 raft_consensus.cc:3037] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.338580 15892 raft_consensus.cc:3037] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:26.341400 15892 raft_consensus.cc:2454] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:26.341400 13235 raft_consensus.cc:2454] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:26.342023 15957 leader_election.cc:302] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:26.343681 17959 raft_consensus.cc:2781] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:26.344082 17959 raft_consensus.cc:686] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:26.344422 17959 consensus_queue.cc:237] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:26.348081 12833 catalog_manager.cc:5484] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:26.374296 13235 consensus_queue.cc:237] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:26.376353 16025 raft_consensus.cc:1261] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:26.376986 12974 raft_consensus.cc:1261] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:26.377552 17962 consensus_queue.cc:1035] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:26.377833 17399 consensus_queue.cc:1035] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:26.380844 17962 raft_consensus.cc:2932] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:26.381438 16025 raft_consensus.cc:2932] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:26.382043 12974 raft_consensus.cc:2932] T 4fc5dbfe3a9b4c84a3217981527e73ea P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:26.387642 12826 catalog_manager.cc:5484] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:26.391711 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fc5dbfe3a9b4c84a3217981527e73ea with delete_type TABLET_DATA_TOMBSTONED (TS e3fefd4dca2d41678187047f06c10187 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:26.392076 17934 tablet_replica.cc:330] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:26.392331 17934 raft_consensus.cc:2229] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:26.392616 17934 raft_consensus.cc:2258] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:26.393491 17934 ts_tablet_manager.cc:1875] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:26.399948 17934 ts_tablet_manager.cc:1888] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:26.400117 17934 log.cc:1196] T 4fc5dbfe3a9b4c84a3217981527e73ea P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/4fc5dbfe3a9b4c84a3217981527e73ea
I20240501 08:12:26.400911 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 4fc5dbfe3a9b4c84a3217981527e73ea (table rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]) successfully deleted
I20240501 08:12:26.450742 12833 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:26.458359 15754 raft_consensus.cc:1261] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:26.459077 16025 raft_consensus.cc:1261] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:26.459703 17963 consensus_queue.cc:1035] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.460049 17472 consensus_queue.cc:1035] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.466862 12833 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet c8f25679cb4c4d69aa2864726088ed3e (table rebalancer_dynamic_table [id=f3a45aa4df9948c3881fd116481daefd]). Expected version 1 got 0
I20240501 08:12:26.502617 18011 tablet.cc:1722] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:26.503613 18011 tablet.cc:1620] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:26.504166 12833 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet c8f25679cb4c4d69aa2864726088ed3e (table rebalancer_dynamic_table [id=f3a45aa4df9948c3881fd116481daefd]). Expected version 1 got 0
I20240501 08:12:26.511327 18010 tablet.cc:1722] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:26.513595 18010 tablet.cc:1620] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:26.514156 12832 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet c8f25679cb4c4d69aa2864726088ed3e (table rebalancer_dynamic_table [id=f3a45aa4df9948c3881fd116481daefd]). Expected version 1 got 0
I20240501 08:12:26.518805 18014 tablet.cc:1722] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:26.527321 12833 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet c8f25679cb4c4d69aa2864726088ed3e (table rebalancer_dynamic_table [id=f3a45aa4df9948c3881fd116481daefd]). Expected version 1 got 0
I20240501 08:12:26.528218 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=f3a45aa4df9948c3881fd116481daefd] alter complete (version 1)
I20240501 08:12:26.519603 18014 tablet.cc:1620] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:26.604465 17831 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.617029 17875 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.644049 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:26.654681 18010 tablet.cc:1722] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:26.655833 18010 tablet.cc:1620] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:26.658208 18014 tablet.cc:1722] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:26.658737 18014 tablet.cc:1620] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:26.660723 18011 tablet.cc:1722] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:26.661228 18011 tablet.cc:1620] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:26.668833 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=f3a45aa4df9948c3881fd116481daefd] alter complete (version 2)
I20240501 08:12:26.699047 15872 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:26.723934 13345 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:26.751591 16003 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:26.773815 16138 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 08:12:26.780401 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:26.788059 15736 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:26.795361 18010 tablet.cc:1722] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:26.800398 18011 tablet.cc:1722] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:26.806998 18014 tablet.cc:1722] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:26.812124 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=f3a45aa4df9948c3881fd116481daefd] alter complete (version 3)
I20240501 08:12:26.827883 17876 consensus_queue.cc:1035] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.838652 12954 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:26.839743 13213 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:26.845866 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:26.866719 17875 consensus_queue.cc:1035] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.900499 17959 consensus_queue.cc:1035] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.905071 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:26.919374 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 7a71b0688ca04973a46b3cbc2ce680d2 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.919564 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 7a71b0688ca04973a46b3cbc2ce680d2 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.919675 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 7a71b0688ca04973a46b3cbc2ce680d2 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.919914 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a71b0688ca04973a46b3cbc2ce680d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.919962 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a71b0688ca04973a46b3cbc2ce680d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.920125 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a71b0688ca04973a46b3cbc2ce680d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.921582 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 2f795db6dcb644d691c3ba9414a62c51 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.922016 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f795db6dcb644d691c3ba9414a62c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.922214 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 2f795db6dcb644d691c3ba9414a62c51 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.922598 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f795db6dcb644d691c3ba9414a62c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.925169 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 5395a66a354241dbb5efdaa6f68f50c1 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.925194 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 2f795db6dcb644d691c3ba9414a62c51 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.925629 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f795db6dcb644d691c3ba9414a62c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.925710 16136 tablet_service.cc:1467] Processing CreateTablet for tablet 5395a66a354241dbb5efdaa6f68f50c1 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.926133 16136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5395a66a354241dbb5efdaa6f68f50c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.929961 12833 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:26.930245 12833 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:26.930789 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 5395a66a354241dbb5efdaa6f68f50c1 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:26.931293 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5395a66a354241dbb5efdaa6f68f50c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.934826 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5395a66a354241dbb5efdaa6f68f50c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:26.941241 12833 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet c8f25679cb4c4d69aa2864726088ed3e
I20240501 08:12:26.942025 16004 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f25679cb4c4d69aa2864726088ed3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:26 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:26.944834 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f25679cb4c4d69aa2864726088ed3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:26 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:26.945453 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f25679cb4c4d69aa2864726088ed3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:26 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:26.951380 17959 consensus_queue.cc:1035] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:26.970131 18059 tablet_bootstrap.cc:492] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:26.970710 18061 tablet_bootstrap.cc:492] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:26.976265 18057 tablet_bootstrap.cc:492] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:26.981860 18056 tablet_bootstrap.cc:492] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:26.984364 18056 tablet_bootstrap.cc:654] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.984928 18055 tablet_replica.cc:330] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:26.989267 18057 tablet_bootstrap.cc:654] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.989444 18060 tablet_bootstrap.cc:492] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:26.990312 18053 tablet_replica.cc:330] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:26.991163 18059 tablet_bootstrap.cc:654] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:26.993070 18053 raft_consensus.cc:2229] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:26.993837 18053 raft_consensus.cc:2258] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:26.994372 18055 raft_consensus.cc:2229] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:26.994791 18055 raft_consensus.cc:2258] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.000810 18062 tablet_bootstrap.cc:492] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:27.001112 18060 tablet_bootstrap.cc:654] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.002400 18055 ts_tablet_manager.cc:1875] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:27.002935 18062 tablet_bootstrap.cc:654] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.004127 18054 tablet_replica.cc:330] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:26.991060 18061 tablet_bootstrap.cc:654] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.004505 18054 raft_consensus.cc:2229] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:27.005205 18054 raft_consensus.cc:2258] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.008312 18061 tablet_bootstrap.cc:492] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:27.008468 18061 ts_tablet_manager.cc:1367] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240501 08:12:27.008879 18057 tablet_bootstrap.cc:492] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:27.009127 18057 ts_tablet_manager.cc:1367] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.001s
I20240501 08:12:27.010154 18057 raft_consensus.cc:348] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:27.010527 18057 raft_consensus.cc:374] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.010716 18057 raft_consensus.cc:729] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.011111 18057 consensus_queue.cc:260] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:27.011871 18057 ts_tablet_manager.cc:1398] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:27.012279 18057 tablet_bootstrap.cc:492] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:27.012706 18061 raft_consensus.cc:348] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.012939 18061 raft_consensus.cc:374] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.013038 18061 raft_consensus.cc:729] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.013307 18061 consensus_queue.cc:260] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.013891 18061 ts_tablet_manager.cc:1398] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:27.019397 18053 ts_tablet_manager.cc:1875] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:27.030822 18059 tablet_bootstrap.cc:492] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:27.031087 18059 ts_tablet_manager.cc:1367] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20240501 08:12:27.031987 18059 raft_consensus.cc:348] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:27.032215 18059 raft_consensus.cc:374] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.032371 18059 raft_consensus.cc:729] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.032707 18059 consensus_queue.cc:260] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:27.033460 18059 ts_tablet_manager.cc:1398] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:27.033881 18059 tablet_bootstrap.cc:492] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:27.017117 18057 tablet_bootstrap.cc:654] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.035624 18055 ts_tablet_manager.cc:1888] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:27.035770 18055 log.cc:1196] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/c8f25679cb4c4d69aa2864726088ed3e
I20240501 08:12:27.036314 18055 ts_tablet_manager.cc:1909] T c8f25679cb4c4d69aa2864726088ed3e P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:27.037353 18059 tablet_bootstrap.cc:654] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.037544 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet c8f25679cb4c4d69aa2864726088ed3e (table rebalancer_dynamic_table_new_name [id=f3a45aa4df9948c3881fd116481daefd]) successfully deleted
I20240501 08:12:27.037799 18054 ts_tablet_manager.cc:1875] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:27.041805 18062 tablet_bootstrap.cc:492] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:27.043063 18060 tablet_bootstrap.cc:492] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:27.044772 18060 ts_tablet_manager.cc:1367] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.002s
I20240501 08:12:27.047101 12826 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:27.048439 18060 raft_consensus.cc:348] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:27.046272 18062 ts_tablet_manager.cc:1367] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.045s user 0.001s sys 0.003s
I20240501 08:12:27.053531 18060 raft_consensus.cc:374] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.053663 18060 raft_consensus.cc:729] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.053684 18062 raft_consensus.cc:348] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:27.054006 18062 raft_consensus.cc:374] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.054121 18062 raft_consensus.cc:729] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.054271 18060 consensus_queue.cc:260] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:27.054411 18062 consensus_queue.cc:260] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:27.055403 18060 ts_tablet_manager.cc:1398] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 08:12:27.055600 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8ddc9f7952194ff7afba2166c8e7cf3f]), partition=
I20240501 08:12:27.056015 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.056241 16136 tablet_service.cc:1467] Processing CreateTablet for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8ddc9f7952194ff7afba2166c8e7cf3f]), partition=
I20240501 08:12:27.056628 16136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.060735 17927 raft_consensus.cc:482] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.060874 17927 raft_consensus.cc:504] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:27.061651 17927 leader_election.cc:288] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:27.062778 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5395a66a354241dbb5efdaa6f68f50c1" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:27.063031 13104 raft_consensus.cc:2454] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:27.063517 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5395a66a354241dbb5efdaa6f68f50c1" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:27.063743 15893 raft_consensus.cc:2454] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:27.064010 18054 ts_tablet_manager.cc:1888] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:27.064129 18054 log.cc:1196] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/c8f25679cb4c4d69aa2864726088ed3e
I20240501 08:12:27.064827 16092 leader_election.cc:302] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:27.065452 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8ddc9f7952194ff7afba2166c8e7cf3f]), partition=
I20240501 08:12:27.065768 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.067205 17927 raft_consensus.cc:2781] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:27.067313 17927 raft_consensus.cc:482] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:27.067314 18054 ts_tablet_manager.cc:1909] T c8f25679cb4c4d69aa2864726088ed3e P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:27.067401 17927 raft_consensus.cc:3037] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.070325 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet c8f25679cb4c4d69aa2864726088ed3e (table rebalancer_dynamic_table_new_name [id=f3a45aa4df9948c3881fd116481daefd]) successfully deleted
I20240501 08:12:27.074220 17927 raft_consensus.cc:504] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:27.075735 17927 leader_election.cc:288] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:27.076270 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5395a66a354241dbb5efdaa6f68f50c1" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:27.076572 15893 raft_consensus.cc:3037] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.079481 15893 raft_consensus.cc:2454] T 5395a66a354241dbb5efdaa6f68f50c1 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:27.081727 16090 leader_election.cc:302] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:27.083597 18053 ts_tablet_manager.cc:1888] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:27.084172 18062 ts_tablet_manager.cc:1398] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240501 08:12:27.086266 18053 log.cc:1196] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/c8f25679cb4c4d69aa2864726088ed3e
I20240501 08:12:27.086970 18053 ts_tablet_manager.cc:1909] T c8f25679cb4c4d69aa2864726088ed3e P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:27.087607 18057 tablet_bootstrap.cc:492] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:27.087816 18057 ts_tablet_manager.cc:1367] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20240501 08:12:27.089355 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet c8f25679cb4c4d69aa2864726088ed3e (table rebalancer_dynamic_table_new_name [id=f3a45aa4df9948c3881fd116481daefd]) successfully deleted
I20240501 08:12:27.090001 18057 raft_consensus.cc:348] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:27.090291 18057 raft_consensus.cc:374] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.090371 18057 raft_consensus.cc:729] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.090610 18057 consensus_queue.cc:260] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:27.090974 18060 tablet_bootstrap.cc:492] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:27.091184 18057 ts_tablet_manager.cc:1398] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:27.092648 18060 tablet_bootstrap.cc:654] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.095274 18059 tablet_bootstrap.cc:492] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:27.095280 18029 raft_consensus.cc:482] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.095468 18056 tablet_bootstrap.cc:492] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:27.095466 18029 raft_consensus.cc:504] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:27.095881 18056 ts_tablet_manager.cc:1367] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.114s user 0.014s sys 0.010s
I20240501 08:12:27.096261 18029 leader_election.cc:288] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:27.096855 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f795db6dcb644d691c3ba9414a62c51" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:27.097050 18056 raft_consensus.cc:348] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:27.097333 18056 raft_consensus.cc:374] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.097844 18056 raft_consensus.cc:729] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.098253 18059 ts_tablet_manager.cc:1367] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.000s
I20240501 08:12:27.098215 18056 consensus_queue.cc:260] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:27.098652 15755 raft_consensus.cc:2454] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:27.099416 13300 leader_election.cc:302] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:27.099570 18059 raft_consensus.cc:348] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.099743 18029 raft_consensus.cc:2781] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:27.099788 18059 raft_consensus.cc:374] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.099843 18029 raft_consensus.cc:482] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:27.099860 18059 raft_consensus.cc:729] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.099925 18029 raft_consensus.cc:3037] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.100103 18059 consensus_queue.cc:260] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.100739 18059 ts_tablet_manager.cc:1398] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:27.103255 18029 raft_consensus.cc:504] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:27.103549 18064 raft_consensus.cc:482] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.103688 18064 raft_consensus.cc:504] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.104488 18064 leader_election.cc:288] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:27.105224 18057 tablet_bootstrap.cc:492] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:27.105386 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f795db6dcb644d691c3ba9414a62c51" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:27.105612 15755 raft_consensus.cc:3037] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.107146 18057 tablet_bootstrap.cc:654] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.108049 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a71b0688ca04973a46b3cbc2ce680d2" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
W20240501 08:12:27.108871 15957 leader_election.cc:341] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:27.109185 18060 tablet_bootstrap.cc:492] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:27.109354 18060 ts_tablet_manager.cc:1367] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.003s
I20240501 08:12:27.109490 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a71b0688ca04973a46b3cbc2ce680d2" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:27.109746 13104 raft_consensus.cc:2454] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:27.110177 15959 leader_election.cc:302] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters: 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:27.110613 18064 raft_consensus.cc:2781] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:27.110734 18064 raft_consensus.cc:482] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:27.110841 18064 raft_consensus.cc:3037] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.113672 18064 raft_consensus.cc:504] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.113991 15755 raft_consensus.cc:2454] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:27.114740 13300 leader_election.cc:302] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:27.115139 17689 raft_consensus.cc:2781] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:27.115273 17689 raft_consensus.cc:686] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:27.115556 17689 consensus_queue.cc:237] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:27.117460 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a71b0688ca04973a46b3cbc2ce680d2" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:27.117689 13104 raft_consensus.cc:3037] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.118458 18064 leader_election.cc:288] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:27.119016 18060 raft_consensus.cc:348] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.119256 18060 raft_consensus.cc:374] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.119319 18060 raft_consensus.cc:729] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.119594 18060 consensus_queue.cc:260] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.124444 13104 raft_consensus.cc:2454] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:27.125023 15959 leader_election.cc:302] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:27.125196 18060 ts_tablet_manager.cc:1398] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240501 08:12:27.125413 18064 raft_consensus.cc:2781] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:27.125458 18029 leader_election.cc:288] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:27.126009 18064 raft_consensus.cc:686] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:27.126375 18064 consensus_queue.cc:237] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.128728 18074 raft_consensus.cc:2781] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:27.128911 18074 raft_consensus.cc:686] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:27.129249 18074 consensus_queue.cc:237] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
W20240501 08:12:27.130685 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.69:42897, user_credentials={real_user=slave}} blocked reactor thread for 63779.4us
I20240501 08:12:27.131155 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f795db6dcb644d691c3ba9414a62c51" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:27.131366 16153 raft_consensus.cc:2454] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:27.131337 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5395a66a354241dbb5efdaa6f68f50c1" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:27.131513 13104 raft_consensus.cc:3037] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.132820 18056 ts_tablet_manager.cc:1398] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240501 08:12:27.133438 18056 tablet_bootstrap.cc:492] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:27.133893 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f795db6dcb644d691c3ba9414a62c51" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:27.134093 16153 raft_consensus.cc:3037] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.134546 13104 raft_consensus.cc:2454] T 5395a66a354241dbb5efdaa6f68f50c1 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:27.137037 16153 raft_consensus.cc:2454] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:27.138908 18056 tablet_bootstrap.cc:654] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.143384 12831 catalog_manager.cc:5484] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:27.144055 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a71b0688ca04973a46b3cbc2ce680d2" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
W20240501 08:12:27.146618 15957 leader_election.cc:341] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:27.148274 12826 catalog_manager.cc:5484] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:27.158457 12826 catalog_manager.cc:5484] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:27.160799 18056 tablet_bootstrap.cc:492] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:27.161289 18056 ts_tablet_manager.cc:1367] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.002s
I20240501 08:12:27.162199 18056 raft_consensus.cc:348] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.162533 18056 raft_consensus.cc:374] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.162755 18056 raft_consensus.cc:729] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.163113 18056 consensus_queue.cc:260] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.164733 18056 ts_tablet_manager.cc:1398] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:27.167192 18057 tablet_bootstrap.cc:492] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:27.167392 18057 ts_tablet_manager.cc:1367] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.001s
I20240501 08:12:27.168468 18057 raft_consensus.cc:348] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.168746 18057 raft_consensus.cc:374] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.168861 18057 raft_consensus.cc:729] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.169202 18057 consensus_queue.cc:260] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.169939 18057 ts_tablet_manager.cc:1398] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:27.170233 18057 tablet_bootstrap.cc:492] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:27.172235 18057 tablet_bootstrap.cc:654] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.179266 18057 tablet_bootstrap.cc:492] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:27.183548 18057 ts_tablet_manager.cc:1367] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 08:12:27.184620 18057 raft_consensus.cc:348] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.184952 18057 raft_consensus.cc:374] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.185057 18057 raft_consensus.cc:729] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.185329 18057 consensus_queue.cc:260] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.185950 18057 ts_tablet_manager.cc:1398] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:27.263653 13364 consensus_queue.cc:237] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.265785 13231 raft_consensus.cc:1261] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:27.265918 16153 raft_consensus.cc:1261] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:27.266341 17689 consensus_queue.cc:1035] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:27.266613 17942 consensus_queue.cc:1035] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:27.269215 18029 raft_consensus.cc:2932] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:27.269820 13231 raft_consensus.cc:2932] T 3b5240d5db124f409b28fbde8cf68f8a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:27.269807 16153 raft_consensus.cc:2932] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:27.275136 12826 catalog_manager.cc:5484] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:27.283157 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5240d5db124f409b28fbde8cf68f8a with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:27.286276 18104 tablet_replica.cc:330] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:27.286517 18104 raft_consensus.cc:2229] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:27.286810 18104 raft_consensus.cc:2258] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:27.287425 18104 ts_tablet_manager.cc:1875] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:27.293591 18104 ts_tablet_manager.cc:1888] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:27.294016 18104 log.cc:1196] T 3b5240d5db124f409b28fbde8cf68f8a P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/3b5240d5db124f409b28fbde8cf68f8a
I20240501 08:12:27.294900 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 3b5240d5db124f409b28fbde8cf68f8a (table rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]) successfully deleted
I20240501 08:12:27.403136 18074 raft_consensus.cc:482] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.403375 18074 raft_consensus.cc:504] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.404140 18074 leader_election.cc:288] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:27.404918 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ca3fec562b4a2a8a3eb21a7a6fa042" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:27.405162 15893 raft_consensus.cc:2454] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:27.405144 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ca3fec562b4a2a8a3eb21a7a6fa042" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:27.405376 15755 raft_consensus.cc:2454] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:27.407171 16090 leader_election.cc:302] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:27.408179 18074 raft_consensus.cc:2781] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:27.408277 18074 raft_consensus.cc:482] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:27.408412 18074 raft_consensus.cc:3037] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.411244 18074 raft_consensus.cc:504] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.415980 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ca3fec562b4a2a8a3eb21a7a6fa042" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:27.416225 15893 raft_consensus.cc:3037] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.420897 15893 raft_consensus.cc:2454] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:27.421350 16090 leader_election.cc:302] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:27.421780 18076 raft_consensus.cc:2781] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:27.421960 18076 raft_consensus.cc:686] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:27.422277 18076 consensus_queue.cc:237] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.428474 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45ca3fec562b4a2a8a3eb21a7a6fa042" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:27.428720 15755 raft_consensus.cc:3037] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.431510 15755 raft_consensus.cc:2454] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:27.432680 18074 leader_election.cc:288] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:27.436846 12829 catalog_manager.cc:5484] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:27.461972 15870 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:27.470202 13345 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:27.497732 16004 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:27.526888 18076 consensus_queue.cc:1035] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.529827 16136 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:27.546428 18074 raft_consensus.cc:482] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.546626 18074 raft_consensus.cc:504] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.547305 18074 leader_election.cc:288] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:27.548153 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a71b0688ca04973a46b3cbc2ce680d2" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:27.548444 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a71b0688ca04973a46b3cbc2ce680d2" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:27.548717 13104 raft_consensus.cc:2379] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cc32f6e2d49477f9a23ce6e444039fb in current term 1: Already voted for candidate 5dd722c770d34d6c9154a87917117161 in this term.
I20240501 08:12:27.549165 16092 leader_election.cc:302] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187
I20240501 08:12:27.549458 18074 raft_consensus.cc:3037] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.552631 18074 raft_consensus.cc:2726] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:27.568939 12829 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:27.570395 18074 consensus_queue.cc:1035] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.592442 17942 consensus_queue.cc:1035] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.598461 13213 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:27.623848 15754 raft_consensus.cc:1261] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:27.624711 18076 consensus_queue.cc:1035] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.642503 15892 raft_consensus.cc:1261] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:27.646201 18076 consensus_queue.cc:1035] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 08:12:27.648753 18029 consensus_queue.cc:1035] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.668217 12829 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (table rebalancer_dynamic_table [id=8ddc9f7952194ff7afba2166c8e7cf3f]). Expected version 1 got 0
I20240501 08:12:27.675127 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:27.675770 12829 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (table rebalancer_dynamic_table [id=8ddc9f7952194ff7afba2166c8e7cf3f]). Expected version 1 got 0
I20240501 08:12:27.683625 17959 consensus_queue.cc:1035] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.702538 12954 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:27.705672 12831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:27.719029 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 65a3d07808414c50a330da007c8f5e19 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.719537 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a3d07808414c50a330da007c8f5e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.732820 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 65a3d07808414c50a330da007c8f5e19 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
W20240501 08:12:27.733031 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.69:42897, user_credentials={real_user=slave}} blocked reactor thread for 54649.2us
I20240501 08:12:27.733294 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a3d07808414c50a330da007c8f5e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.736860 16004 tablet_service.cc:1467] Processing CreateTablet for tablet b320e884c36d461ba5f2941d9d76185f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.737356 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b320e884c36d461ba5f2941d9d76185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.745177 13084 tablet_service.cc:1467] Processing CreateTablet for tablet b320e884c36d461ba5f2941d9d76185f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.745703 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b320e884c36d461ba5f2941d9d76185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.751417 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 315d149a26c14982aae10b5fb800dc82 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.752048 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315d149a26c14982aae10b5fb800dc82. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:12:27.760766 13301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 56531.4us
I20240501 08:12:27.762760 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 315d149a26c14982aae10b5fb800dc82 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.763254 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315d149a26c14982aae10b5fb800dc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.775130 18177 tablet.cc:1722] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:27.775727 18177 tablet.cc:1620] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:27.777479 18178 tablet_bootstrap.cc:492] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:27.782640 18178 tablet_bootstrap.cc:654] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.782923 12833 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (table rebalancer_dynamic_table [id=8ddc9f7952194ff7afba2166c8e7cf3f]). Expected version 1 got 0
I20240501 08:12:27.784395 18181 tablet_bootstrap.cc:492] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:27.785452 15735 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:27.788403 18179 tablet_bootstrap.cc:492] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:27.791257 18181 tablet_bootstrap.cc:654] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.804332 18179 tablet_bootstrap.cc:654] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.808734 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=8ddc9f7952194ff7afba2166c8e7cf3f] alter complete (version 1)
I20240501 08:12:27.809077 18181 tablet_bootstrap.cc:492] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:27.809669 18181 ts_tablet_manager.cc:1367] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20240501 08:12:27.811736 16136 tablet_service.cc:1467] Processing CreateTablet for tablet 65a3d07808414c50a330da007c8f5e19 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.811436 18181 raft_consensus.cc:348] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.814232 16136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a3d07808414c50a330da007c8f5e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.817368 18181 raft_consensus.cc:374] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.817780 18181 raft_consensus.cc:729] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.818423 18181 consensus_queue.cc:260] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.819248 18180 tablet_bootstrap.cc:492] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:27.820591 18181 ts_tablet_manager.cc:1398] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 08:12:27.823143 18181 tablet_bootstrap.cc:492] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:27.823817 18180 tablet_bootstrap.cc:654] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.830056 18176 tablet.cc:1722] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:27.830686 18176 tablet.cc:1620] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:27.830739 18188 tablet_bootstrap.cc:492] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:27.832377 18178 tablet_bootstrap.cc:492] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:27.832567 18180 tablet_bootstrap.cc:492] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:27.832607 18178 ts_tablet_manager.cc:1367] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.006s
I20240501 08:12:27.833302 18180 ts_tablet_manager.cc:1367] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240501 08:12:27.833966 18178 raft_consensus.cc:348] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.834270 18178 raft_consensus.cc:374] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.834424 18178 raft_consensus.cc:729] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.834743 18178 consensus_queue.cc:260] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.835038 18180 raft_consensus.cc:348] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
W20240501 08:12:27.835480 16090 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 79840.6us
I20240501 08:12:27.835714 18180 raft_consensus.cc:374] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.836198 18180 raft_consensus.cc:729] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.835995 17959 consensus_queue.cc:1035] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:27.840564 18178 ts_tablet_manager.cc:1398] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 08:12:27.848922 18188 tablet_bootstrap.cc:654] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.836719 18180 consensus_queue.cc:260] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.853521 18181 tablet_bootstrap.cc:654] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.856989 18190 tablet.cc:1722] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:27.858019 18190 tablet.cc:1620] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:27.869015 18180 ts_tablet_manager.cc:1398] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240501 08:12:27.869570 18180 tablet_bootstrap.cc:492] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:27.871866 18180 tablet_bootstrap.cc:654] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.873548 16136 tablet_service.cc:1467] Processing CreateTablet for tablet b320e884c36d461ba5f2941d9d76185f (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.873991 16136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b320e884c36d461ba5f2941d9d76185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:27.877074 18179 tablet_bootstrap.cc:492] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:27.877264 18179 ts_tablet_manager.cc:1367] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.004s
I20240501 08:12:27.878079 18179 raft_consensus.cc:348] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.879248 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 315d149a26c14982aae10b5fb800dc82 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:27.879671 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315d149a26c14982aae10b5fb800dc82. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:12:27.886354 15823 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 48327.1us
I20240501 08:12:27.888315 12833 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (table rebalancer_dynamic_table [id=8ddc9f7952194ff7afba2166c8e7cf3f]). Expected version 1 got 0
I20240501 08:12:27.896414 18179 raft_consensus.cc:374] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.896623 18179 raft_consensus.cc:729] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.896912 18179 consensus_queue.cc:260] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.898139 18179 ts_tablet_manager.cc:1398] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.020s user 0.001s sys 0.002s
I20240501 08:12:27.904948 18180 tablet_bootstrap.cc:492] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:27.905226 18180 ts_tablet_manager.cc:1367] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.006s
I20240501 08:12:27.906152 18180 raft_consensus.cc:348] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.906356 18180 raft_consensus.cc:374] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.907660 18180 raft_consensus.cc:729] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.908459 18180 consensus_queue.cc:260] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.909154 18180 ts_tablet_manager.cc:1398] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:27.913479 18188 tablet_bootstrap.cc:492] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:27.913637 18188 ts_tablet_manager.cc:1367] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.004s
I20240501 08:12:27.914781 18181 tablet_bootstrap.cc:492] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:27.915015 18181 ts_tablet_manager.cc:1367] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.004s
I20240501 08:12:27.915942 18188 raft_consensus.cc:348] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.925218 18188 raft_consensus.cc:374] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.917253 18181 raft_consensus.cc:348] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.925519 18188 raft_consensus.cc:729] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.925657 18181 raft_consensus.cc:374] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.925861 18181 raft_consensus.cc:729] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.926113 18188 consensus_queue.cc:260] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.927225 18181 consensus_queue.cc:260] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.932233 18188 ts_tablet_manager.cc:1398] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.018s user 0.001s sys 0.002s
I20240501 08:12:27.935197 18181 ts_tablet_manager.cc:1398] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240501 08:12:27.937079 18188 tablet_bootstrap.cc:492] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:27.943818 12833 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:27.955768 18188 tablet_bootstrap.cc:654] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:27.958520 18176 tablet.cc:1722] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:27.959105 18176 tablet.cc:1620] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:27.966161 18177 tablet.cc:1722] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:27.966642 18177 tablet.cc:1620] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:27.967936 18190 tablet.cc:1722] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:27.968708 18190 tablet.cc:1620] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:27.982090 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=8ddc9f7952194ff7afba2166c8e7cf3f] alter complete (version 2)
I20240501 08:12:27.983108 17959 raft_consensus.cc:482] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:27.983292 17959 raft_consensus.cc:504] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.984099 17959 leader_election.cc:288] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:27.985061 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:27.985272 13103 raft_consensus.cc:2454] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:27.985816 15959 leader_election.cc:302] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:27.986424 17959 raft_consensus.cc:2781] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:27.986543 17959 raft_consensus.cc:482] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:27.986634 17959 raft_consensus.cc:3037] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.990731 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:27.991426 18188 tablet_bootstrap.cc:492] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
W20240501 08:12:27.991500 15957 leader_election.cc:341] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:27.991621 18188 ts_tablet_manager.cc:1367] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.010s
I20240501 08:12:27.992511 18188 raft_consensus.cc:348] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.992730 17959 raft_consensus.cc:504] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.992861 18188 raft_consensus.cc:374] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:27.993072 18188 raft_consensus.cc:729] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:27.993386 18188 consensus_queue.cc:260] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:27.993842 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:27.994078 13104 raft_consensus.cc:3037] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:27.995291 18188 ts_tablet_manager.cc:1398] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:27.995569 18188 tablet_bootstrap.cc:492] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:27.996767 13104 raft_consensus.cc:2454] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:27.997243 15959 leader_election.cc:302] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:27.997538 18064 raft_consensus.cc:2781] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:27.998557 18188 tablet_bootstrap.cc:654] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.000773 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
W20240501 08:12:28.001302 15957 leader_election.cc:341] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:28.001802 17959 leader_election.cc:288] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:28.012992 18064 raft_consensus.cc:686] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:28.013532 18064 consensus_queue.cc:237] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.037292 12829 catalog_manager.cc:5484] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:28.063270 18188 tablet_bootstrap.cc:492] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:28.063447 18188 ts_tablet_manager.cc:1367] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.068s user 0.026s sys 0.029s
I20240501 08:12:28.064311 18188 raft_consensus.cc:348] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.064538 18188 raft_consensus.cc:374] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.064616 18188 raft_consensus.cc:729] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.064956 18188 consensus_queue.cc:260] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.066483 18188 ts_tablet_manager.cc:1398] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:28.090031 12829 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:28.102257 18176 tablet.cc:1722] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:28.104753 18190 tablet.cc:1722] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:28.106659 18177 tablet.cc:1722] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:28.116838 12829 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=8ddc9f7952194ff7afba2166c8e7cf3f] alter complete (version 3)
I20240501 08:12:28.129226 16024 consensus_queue.cc:237] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.129822 18208 raft_consensus.cc:482] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.129967 18208 raft_consensus.cc:504] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.130803 18208 leader_election.cc:288] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:28.131630 12971 raft_consensus.cc:1261] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:28.132198 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315d149a26c14982aae10b5fb800dc82" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:28.132172 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315d149a26c14982aae10b5fb800dc82" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:28.132411 17959 consensus_queue.cc:1035] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:28.132480 15892 raft_consensus.cc:2454] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:28.132474 16153 raft_consensus.cc:2454] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:28.133400 15957 leader_election.cc:302] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:28.133822 17959 raft_consensus.cc:2781] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:28.133944 17959 raft_consensus.cc:482] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:28.134033 17959 raft_consensus.cc:3037] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.134114 15756 raft_consensus.cc:1261] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:28.134709 18064 consensus_queue.cc:1035] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:28.137765 17959 raft_consensus.cc:504] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.138602 17959 leader_election.cc:288] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:28.139112 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315d149a26c14982aae10b5fb800dc82" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:28.139101 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "315d149a26c14982aae10b5fb800dc82" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:28.139333 15892 raft_consensus.cc:3037] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.139372 16153 raft_consensus.cc:3037] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.141330 15892 raft_consensus.cc:2454] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:28.141755 15957 leader_election.cc:302] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:28.142087 17959 raft_consensus.cc:2781] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:28.142246 17959 raft_consensus.cc:686] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:28.142525 17959 consensus_queue.cc:237] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.142736 18064 raft_consensus.cc:2932] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:28.143031 16153 raft_consensus.cc:2454] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:28.146099 15756 raft_consensus.cc:2932] T bdabd94a46dc4a5796949fdf9d0c698c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:28.146689 12830 catalog_manager.cc:5484] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:28.153506 12830 catalog_manager.cc:5484] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:28.154697 12971 raft_consensus.cc:2932] T bdabd94a46dc4a5796949fdf9d0c698c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:28.160835 18174 raft_consensus.cc:482] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.160861 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet bdabd94a46dc4a5796949fdf9d0c698c with delete_type TABLET_DATA_TOMBSTONED (TS 4d516916eeac468db514eeb928f06190 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:28.160995 18174 raft_consensus.cc:504] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.161628 18232 tablet_replica.cc:330] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:28.161708 18174 leader_election.cc:288] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:28.161942 18232 raft_consensus.cc:2229] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:28.162374 18232 raft_consensus.cc:2258] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:28.162921 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a3d07808414c50a330da007c8f5e19" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:28.163199 16153 raft_consensus.cc:2454] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:28.163671 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a3d07808414c50a330da007c8f5e19" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:28.163838 13298 leader_election.cc:302] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:28.163954 15892 raft_consensus.cc:2454] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:28.164127 18174 raft_consensus.cc:2781] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:28.164232 18232 ts_tablet_manager.cc:1875] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:28.164238 18174 raft_consensus.cc:482] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:28.164355 18174 raft_consensus.cc:3037] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.167220 18174 raft_consensus.cc:504] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.167893 18174 leader_election.cc:288] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:28.168496 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a3d07808414c50a330da007c8f5e19" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:28.168735 15892 raft_consensus.cc:3037] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.168678 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65a3d07808414c50a330da007c8f5e19" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:28.168953 16153 raft_consensus.cc:3037] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.171414 15892 raft_consensus.cc:2454] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:28.171530 18232 ts_tablet_manager.cc:1888] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:28.171625 18232 log.cc:1196] T bdabd94a46dc4a5796949fdf9d0c698c P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/bdabd94a46dc4a5796949fdf9d0c698c
I20240501 08:12:28.172082 13298 leader_election.cc:302] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters:
I20240501 08:12:28.172437 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet bdabd94a46dc4a5796949fdf9d0c698c (table rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]) successfully deleted
I20240501 08:12:28.172482 16153 raft_consensus.cc:2454] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:28.172685 18174 raft_consensus.cc:2781] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:28.172878 18174 raft_consensus.cc:686] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:28.173206 18174 consensus_queue.cc:237] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.177490 12829 catalog_manager.cc:5484] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:28.225848 12829 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:28.226146 12829 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:28.229849 12829 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 45ca3fec562b4a2a8a3eb21a7a6fa042
I20240501 08:12:28.230608 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:28 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:28.230943 18232 tablet_replica.cc:330] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:28.231169 18232 raft_consensus.cc:2229] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:28.231328 16136 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:28 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:28.231401 18232 raft_consensus.cc:2258] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:28.231626 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:28 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:28.232698 18242 tablet_replica.cc:330] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:28.232784 18243 tablet_replica.cc:330] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:28.232985 18242 raft_consensus.cc:2229] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:28.233428 18242 raft_consensus.cc:2258] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:28.233827 18243 raft_consensus.cc:2229] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:28.234035 18243 raft_consensus.cc:2258] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:28.234578 18232 ts_tablet_manager.cc:1875] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:28.234867 18243 ts_tablet_manager.cc:1875] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:28.235117 18242 ts_tablet_manager.cc:1875] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:28.242854 18242 ts_tablet_manager.cc:1888] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:28.243089 18242 log.cc:1196] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/45ca3fec562b4a2a8a3eb21a7a6fa042
I20240501 08:12:28.243695 18242 ts_tablet_manager.cc:1909] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:28.244968 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (table rebalancer_dynamic_table_new_name [id=8ddc9f7952194ff7afba2166c8e7cf3f]) successfully deleted
I20240501 08:12:28.246248 18243 ts_tablet_manager.cc:1888] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:28.246387 18232 ts_tablet_manager.cc:1888] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:28.246511 18232 log.cc:1196] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/45ca3fec562b4a2a8a3eb21a7a6fa042
I20240501 08:12:28.246551 18243 log.cc:1196] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/45ca3fec562b4a2a8a3eb21a7a6fa042
I20240501 08:12:28.246965 18232 ts_tablet_manager.cc:1909] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:28.247054 18243 ts_tablet_manager.cc:1909] T 45ca3fec562b4a2a8a3eb21a7a6fa042 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:28.247952 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (table rebalancer_dynamic_table_new_name [id=8ddc9f7952194ff7afba2166c8e7cf3f]) successfully deleted
I20240501 08:12:28.248035 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 45ca3fec562b4a2a8a3eb21a7a6fa042 (table rebalancer_dynamic_table_new_name [id=8ddc9f7952194ff7afba2166c8e7cf3f]) successfully deleted
I20240501 08:12:28.336563 12827 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:28.346813 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 113a070e074040d29beaf231ef6dbe7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c50eaa08a2ec4b31923ef2396c6b1f9d]), partition=
I20240501 08:12:28.347235 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113a070e074040d29beaf231ef6dbe7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.348990 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 113a070e074040d29beaf231ef6dbe7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c50eaa08a2ec4b31923ef2396c6b1f9d]), partition=
I20240501 08:12:28.349391 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113a070e074040d29beaf231ef6dbe7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.350061 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 113a070e074040d29beaf231ef6dbe7c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c50eaa08a2ec4b31923ef2396c6b1f9d]), partition=
I20240501 08:12:28.350410 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 113a070e074040d29beaf231ef6dbe7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.355834 18275 tablet_bootstrap.cc:492] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:28.357764 18275 tablet_bootstrap.cc:654] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.363860 18180 tablet_bootstrap.cc:492] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:28.365005 18275 tablet_bootstrap.cc:492] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:28.365165 18275 ts_tablet_manager.cc:1367] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 08:12:28.365523 18180 tablet_bootstrap.cc:654] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.366267 18275 raft_consensus.cc:348] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.366680 18275 raft_consensus.cc:374] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.366822 18275 raft_consensus.cc:729] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.367141 18275 consensus_queue.cc:260] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.367954 18275 ts_tablet_manager.cc:1398] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:28.368774 18180 tablet_bootstrap.cc:492] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:28.368983 18180 ts_tablet_manager.cc:1367] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:12:28.369860 18276 tablet_bootstrap.cc:492] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:28.371522 18180 raft_consensus.cc:348] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.371691 18180 raft_consensus.cc:374] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.371783 18180 raft_consensus.cc:729] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.372002 18180 consensus_queue.cc:260] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.372107 18276 tablet_bootstrap.cc:654] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.372529 18180 ts_tablet_manager.cc:1398] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:28.382439 18276 tablet_bootstrap.cc:492] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:28.382674 18276 ts_tablet_manager.cc:1367] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20240501 08:12:28.383769 18276 raft_consensus.cc:348] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.384136 18276 raft_consensus.cc:374] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.384316 18276 raft_consensus.cc:729] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.384634 18276 consensus_queue.cc:260] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.386837 18276 ts_tablet_manager.cc:1398] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:28.405541 17678 raft_consensus.cc:482] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.405776 17678 raft_consensus.cc:504] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.406726 17678 leader_election.cc:288] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:28.407047 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113a070e074040d29beaf231ef6dbe7c" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:28.407280 15892 raft_consensus.cc:2454] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:28.407902 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113a070e074040d29beaf231ef6dbe7c" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:28.408159 15756 raft_consensus.cc:2454] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:28.409929 12907 leader_election.cc:302] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:28.410792 17678 raft_consensus.cc:2781] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:28.410928 17678 raft_consensus.cc:482] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:28.411010 17678 raft_consensus.cc:3037] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.413543 17678 raft_consensus.cc:504] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.414660 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113a070e074040d29beaf231ef6dbe7c" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:28.414903 15892 raft_consensus.cc:3037] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.415407 15870 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:28.422389 18064 consensus_queue.cc:1035] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.429428 15892 raft_consensus.cc:2454] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:28.431958 12907 leader_election.cc:302] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:28.432421 17701 raft_consensus.cc:2781] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:28.460335 17678 leader_election.cc:288] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:28.477324 13345 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:12:28.478200 18074 raft_consensus.cc:482] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.478435 18074 raft_consensus.cc:504] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.479287 18074 leader_election.cc:288] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:28.480731 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:28.481441 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:28.483003 16092 leader_election.cc:302] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187
I20240501 08:12:28.483618 18074 raft_consensus.cc:2726] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:28.494385 16003 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:28.504057 16157 raft_consensus.cc:3037] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.512916 18064 consensus_queue.cc:1035] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.589437 13213 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:28.590404 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "113a070e074040d29beaf231ef6dbe7c" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:28.590670 15756 raft_consensus.cc:3037] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.593564 15756 raft_consensus.cc:2454] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:28.624713 17701 raft_consensus.cc:686] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:28.625211 17701 consensus_queue.cc:237] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:28.630586 16136 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 08:12:28.663093 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:28.675252 12954 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:12:28.677505 12827 catalog_manager.cc:5484] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:28.692487 18174 consensus_queue.cc:1035] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.706329 18174 consensus_queue.cc:1035] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.730818 18202 consensus_queue.cc:1035] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.742507 18064 consensus_queue.cc:1035] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.746738 12832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:28.765692 16136 tablet_service.cc:1467] Processing CreateTablet for tablet 3d16de310e7148768967be392b7af2ee (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.766180 16136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d16de310e7148768967be392b7af2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.769078 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 3d16de310e7148768967be392b7af2ee (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.769205 13345 tablet_service.cc:1467] Processing CreateTablet for tablet 3d16de310e7148768967be392b7af2ee (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.769523 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d16de310e7148768967be392b7af2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.769578 13345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d16de310e7148768967be392b7af2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.779583 13344 tablet_service.cc:1467] Processing CreateTablet for tablet f8c3a95d53fd43b08205d0f278841896 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.780026 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c3a95d53fd43b08205d0f278841896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.780391 16138 tablet_service.cc:1467] Processing CreateTablet for tablet f8c3a95d53fd43b08205d0f278841896 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.780778 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c3a95d53fd43b08205d0f278841896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.780977 15870 tablet_service.cc:1467] Processing CreateTablet for tablet f8c3a95d53fd43b08205d0f278841896 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.781414 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c3a95d53fd43b08205d0f278841896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.787478 16003 tablet_service.cc:1467] Processing CreateTablet for tablet b0dccc451745459d8dca19f9ebe2e47f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.787945 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dccc451745459d8dca19f9ebe2e47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.789573 18180 tablet_bootstrap.cc:492] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:28.791210 18180 tablet_bootstrap.cc:654] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.794447 16137 tablet_service.cc:1467] Processing CreateTablet for tablet b0dccc451745459d8dca19f9ebe2e47f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.794654 13343 tablet_service.cc:1467] Processing CreateTablet for tablet b0dccc451745459d8dca19f9ebe2e47f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:28.794976 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dccc451745459d8dca19f9ebe2e47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.795050 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dccc451745459d8dca19f9ebe2e47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:28.804997 18320 tablet_bootstrap.cc:492] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:28.807430 18276 tablet_bootstrap.cc:492] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:28.809314 18276 tablet_bootstrap.cc:654] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.811311 18319 tablet_bootstrap.cc:492] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:28.813457 18319 tablet_bootstrap.cc:654] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.813800 18180 tablet_bootstrap.cc:492] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:28.814004 18320 tablet_bootstrap.cc:654] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.814199 18180 ts_tablet_manager.cc:1367] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240501 08:12:28.815116 18180 raft_consensus.cc:348] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:28.815455 18180 raft_consensus.cc:374] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.815583 18180 raft_consensus.cc:729] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.816174 18322 tablet_bootstrap.cc:492] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:28.816898 18180 consensus_queue.cc:260] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:28.818249 18322 tablet_bootstrap.cc:654] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.819936 18276 tablet_bootstrap.cc:492] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:28.820135 18276 ts_tablet_manager.cc:1367] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240501 08:12:28.820264 12827 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:28.821120 18276 raft_consensus.cc:348] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.821539 18276 raft_consensus.cc:374] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.821673 18276 raft_consensus.cc:729] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.822003 18276 consensus_queue.cc:260] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.822808 18276 ts_tablet_manager.cc:1398] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:28.824472 18180 ts_tablet_manager.cc:1398] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 08:12:28.830770 15735 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:12:28.831256 15691 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 207242us
I20240501 08:12:28.881481 18322 tablet_bootstrap.cc:492] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:28.881834 18322 ts_tablet_manager.cc:1367] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.015s
I20240501 08:12:28.882730 18322 raft_consensus.cc:348] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:28.883358 18322 raft_consensus.cc:374] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.883606 18322 raft_consensus.cc:729] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.884009 18322 consensus_queue.cc:260] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:28.885378 18322 ts_tablet_manager.cc:1398] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 08:12:28.899427 15892 raft_consensus.cc:1261] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:28.899472 15755 raft_consensus.cc:1261] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:28.900192 17701 consensus_queue.cc:1035] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.900521 18295 consensus_queue.cc:1035] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:28.914012 12827 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 113a070e074040d29beaf231ef6dbe7c (table rebalancer_dynamic_table [id=c50eaa08a2ec4b31923ef2396c6b1f9d]). Expected version 1 got 0
I20240501 08:12:28.914474 18332 tablet.cc:1722] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:28.916393 18332 tablet.cc:1620] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:28.918776 12826 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 113a070e074040d29beaf231ef6dbe7c (table rebalancer_dynamic_table [id=c50eaa08a2ec4b31923ef2396c6b1f9d]). Expected version 1 got 0
I20240501 08:12:28.925132 18319 tablet_bootstrap.cc:492] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:28.925463 18319 ts_tablet_manager.cc:1367] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.114s user 0.005s sys 0.000s
I20240501 08:12:28.926635 18319 raft_consensus.cc:348] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:28.926978 18319 raft_consensus.cc:374] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.927145 18319 raft_consensus.cc:729] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.927484 18319 consensus_queue.cc:260] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:28.931061 18334 tablet.cc:1722] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:28.931664 18334 tablet.cc:1620] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:28.949040 18333 tablet.cc:1722] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:28.950062 18184 raft_consensus.cc:482] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.950239 18184 raft_consensus.cc:504] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:28.950785 18320 tablet_bootstrap.cc:492] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:28.950944 18320 ts_tablet_manager.cc:1367] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.146s user 0.011s sys 0.035s
I20240501 08:12:28.950942 18184 leader_election.cc:288] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:28.951189 18333 tablet.cc:1620] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:28.951694 18320 raft_consensus.cc:348] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:28.951891 18320 raft_consensus.cc:374] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.951895 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c3a95d53fd43b08205d0f278841896" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:28.952136 15893 raft_consensus.cc:2454] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:28.952713 16090 leader_election.cc:302] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:28.952721 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c3a95d53fd43b08205d0f278841896" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:28.952972 18320 raft_consensus.cc:729] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.953009 18184 raft_consensus.cc:2781] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:28.953115 18184 raft_consensus.cc:482] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:28.953208 18184 raft_consensus.cc:3037] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.953230 12827 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=c50eaa08a2ec4b31923ef2396c6b1f9d] alter complete (version 1)
I20240501 08:12:28.953346 18320 consensus_queue.cc:260] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:28.953619 13363 raft_consensus.cc:2454] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:28.954525 18320 ts_tablet_manager.cc:1398] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:12:28.954787 18320 tablet_bootstrap.cc:492] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:28.956135 18319 ts_tablet_manager.cc:1398] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240501 08:12:28.956445 18320 tablet_bootstrap.cc:654] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.956820 18319 tablet_bootstrap.cc:492] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:28.958130 18184 raft_consensus.cc:504] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:28.959872 18319 tablet_bootstrap.cc:654] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.960855 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c3a95d53fd43b08205d0f278841896" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:28.961088 15893 raft_consensus.cc:3037] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.962258 12826 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 113a070e074040d29beaf231ef6dbe7c (table rebalancer_dynamic_table [id=c50eaa08a2ec4b31923ef2396c6b1f9d]). Expected version 1 got 0
I20240501 08:12:28.963513 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8c3a95d53fd43b08205d0f278841896" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:28.963733 13363 raft_consensus.cc:3037] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:28.964268 15893 raft_consensus.cc:2454] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:28.964499 18320 tablet_bootstrap.cc:492] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:28.964702 18320 ts_tablet_manager.cc:1367] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 08:12:28.965656 18320 raft_consensus.cc:348] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.965912 18320 raft_consensus.cc:374] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:28.966017 18320 raft_consensus.cc:729] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:28.966282 18320 consensus_queue.cc:260] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:28.966753 13363 raft_consensus.cc:2454] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:28.966969 18320 ts_tablet_manager.cc:1398] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:28.967242 18320 tablet_bootstrap.cc:492] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:28.968835 18320 tablet_bootstrap.cc:654] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:28.978708 18064 raft_consensus.cc:482] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:28.978968 18064 raft_consensus.cc:504] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:28.979982 18064 leader_election.cc:288] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:28.964818 16090 leader_election.cc:302] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:28.992422 18074 raft_consensus.cc:2781] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:28.965003 18184 leader_election.cc:288] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:28.992836 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dccc451745459d8dca19f9ebe2e47f" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
W20240501 08:12:28.994683 15957 leader_election.cc:341] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:28.996397 18074 raft_consensus.cc:686] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:28.996714 18074 consensus_queue.cc:237] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:29.001520 12829 catalog_manager.cc:5484] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:29.034525 18319 tablet_bootstrap.cc:492] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:29.034801 18319 ts_tablet_manager.cc:1367] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.004s
I20240501 08:12:29.036674 18319 raft_consensus.cc:348] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.039673 18320 tablet_bootstrap.cc:492] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:29.039853 18320 ts_tablet_manager.cc:1367] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.073s user 0.021s sys 0.025s
I20240501 08:12:29.040033 18319 raft_consensus.cc:374] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.040666 18319 raft_consensus.cc:729] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.040997 18320 raft_consensus.cc:348] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:29.041200 18320 raft_consensus.cc:374] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.041280 18320 raft_consensus.cc:729] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.041483 18320 consensus_queue.cc:260] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:29.041852 18319 consensus_queue.cc:260] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.042006 18320 ts_tablet_manager.cc:1398] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:29.045512 18319 ts_tablet_manager.cc:1398] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 08:12:29.046324 18319 tablet_bootstrap.cc:492] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:29.048031 18319 tablet_bootstrap.cc:654] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.054574 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dccc451745459d8dca19f9ebe2e47f" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:29.054797 13363 raft_consensus.cc:2454] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:29.055994 15959 leader_election.cc:302] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 5dd722c770d34d6c9154a87917117161; no voters: 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:29.056262 18202 raft_consensus.cc:2781] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.056463 18202 raft_consensus.cc:482] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.056571 18202 raft_consensus.cc:3037] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.056439 12829 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:29.059752 18202 raft_consensus.cc:504] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:29.060942 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dccc451745459d8dca19f9ebe2e47f" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
W20240501 08:12:29.061481 15957 leader_election.cc:341] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:29.068917 18332 tablet.cc:1722] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:29.069466 18332 tablet.cc:1620] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:29.070927 18334 tablet.cc:1722] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:29.071368 18334 tablet.cc:1620] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:29.077009 18333 tablet.cc:1722] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:29.078814 18333 tablet.cc:1620] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:29.094709 18314 raft_consensus.cc:482] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.095114 18314 raft_consensus.cc:504] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.095824 18314 leader_election.cc:288] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:29.096596 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d16de310e7148768967be392b7af2ee" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:29.096860 16157 raft_consensus.cc:2454] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:29.097370 13298 leader_election.cc:302] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:29.098129 18174 raft_consensus.cc:2781] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.098284 18174 raft_consensus.cc:482] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.098385 18174 raft_consensus.cc:3037] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.098829 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d16de310e7148768967be392b7af2ee" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:29.099025 15755 raft_consensus.cc:2454] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:29.102125 12827 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=c50eaa08a2ec4b31923ef2396c6b1f9d] alter complete (version 2)
I20240501 08:12:29.103005 18174 raft_consensus.cc:504] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.109133 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d16de310e7148768967be392b7af2ee" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:29.109274 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d16de310e7148768967be392b7af2ee" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:29.109359 16157 raft_consensus.cc:3037] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.109464 15755 raft_consensus.cc:3037] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.112566 18174 leader_election.cc:288] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:29.113792 15755 raft_consensus.cc:2454] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:29.113826 18202 leader_election.cc:288] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:29.112272 16157 raft_consensus.cc:2454] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:29.114427 13300 leader_election.cc:302] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters:
W20240501 08:12:29.114737 15959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 58740.9us
I20240501 08:12:29.114997 18174 raft_consensus.cc:2781] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:29.115562 18174 raft_consensus.cc:686] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:29.115947 18174 consensus_queue.cc:237] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.116160 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dccc451745459d8dca19f9ebe2e47f" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:29.116396 13363 raft_consensus.cc:3037] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.119513 13363 raft_consensus.cc:2454] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:29.122275 15959 leader_election.cc:302] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 5dd722c770d34d6c9154a87917117161; no voters: 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:29.122793 18064 raft_consensus.cc:2781] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:29.125161 18319 tablet_bootstrap.cc:492] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:29.125356 18319 ts_tablet_manager.cc:1367] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.079s user 0.000s sys 0.004s
I20240501 08:12:29.126338 18319 raft_consensus.cc:348] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:29.126583 18319 raft_consensus.cc:374] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.126674 18319 raft_consensus.cc:729] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.126931 18319 consensus_queue.cc:260] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:29.127648 18319 ts_tablet_manager.cc:1398] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:29.139780 18064 raft_consensus.cc:686] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:29.140064 12829 catalog_manager.cc:5484] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:29.143854 18064 consensus_queue.cc:237] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:29.150707 12826 catalog_manager.cc:5484] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:29.180974 13363 consensus_queue.cc:237] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:29.182911 15893 raft_consensus.cc:1261] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:29.183212 13104 raft_consensus.cc:1261] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:29.183656 18174 consensus_queue.cc:1035] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:29.184034 18314 consensus_queue.cc:1035] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:29.186364 18310 raft_consensus.cc:2932] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:29.187016 13104 raft_consensus.cc:2932] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:29.187037 15893 raft_consensus.cc:2932] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:29.194870 12829 catalog_manager.cc:5484] T a54efcd41aeb4c24adfa51bb543d9661 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:29.200433 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet a54efcd41aeb4c24adfa51bb543d9661 with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:29.202203 18366 tablet_replica.cc:330] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:29.202545 18366 raft_consensus.cc:2229] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:29.204562 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:29.209275 18366 raft_consensus.cc:2258] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:29.210256 18366 ts_tablet_manager.cc:1875] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:29.213161 18332 tablet.cc:1722] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:29.217077 18366 ts_tablet_manager.cc:1888] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:29.217288 18366 log.cc:1196] T a54efcd41aeb4c24adfa51bb543d9661 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/a54efcd41aeb4c24adfa51bb543d9661
I20240501 08:12:29.217406 18333 tablet.cc:1722] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:29.218482 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet a54efcd41aeb4c24adfa51bb543d9661 (table rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]) successfully deleted
I20240501 08:12:29.220000 18334 tablet.cc:1722] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:29.228911 12827 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=c50eaa08a2ec4b31923ef2396c6b1f9d] alter complete (version 3)
I20240501 08:12:29.335433 12830 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:29.335701 12830 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:29.340878 12830 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 113a070e074040d29beaf231ef6dbe7c
I20240501 08:12:29.342092 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet 113a070e074040d29beaf231ef6dbe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:29 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:29.342705 18401 tablet_replica.cc:330] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:29.343065 18401 raft_consensus.cc:2229] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:29.343116 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet 113a070e074040d29beaf231ef6dbe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:29 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:29.343381 18401 raft_consensus.cc:2258] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:29.344113 18402 tablet_replica.cc:330] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:29.344336 18402 raft_consensus.cc:2229] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:29.344475 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 113a070e074040d29beaf231ef6dbe7c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:29 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:29.344655 18402 raft_consensus.cc:2258] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:29.344660 18401 ts_tablet_manager.cc:1875] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:29.350291 18402 ts_tablet_manager.cc:1875] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:29.368388 18401 ts_tablet_manager.cc:1888] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:29.368556 18401 log.cc:1196] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/113a070e074040d29beaf231ef6dbe7c
I20240501 08:12:29.369035 18401 ts_tablet_manager.cc:1909] T 113a070e074040d29beaf231ef6dbe7c P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:29.370182 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 113a070e074040d29beaf231ef6dbe7c (table rebalancer_dynamic_table_new_name [id=c50eaa08a2ec4b31923ef2396c6b1f9d]) successfully deleted
I20240501 08:12:29.371429 18403 tablet_replica.cc:330] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:29.371748 18403 raft_consensus.cc:2229] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:29.375530 18403 raft_consensus.cc:2258] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:29.376598 18403 ts_tablet_manager.cc:1875] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:29.373648 18402 ts_tablet_manager.cc:1888] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:29.378542 18402 log.cc:1196] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/113a070e074040d29beaf231ef6dbe7c
I20240501 08:12:29.378962 18402 ts_tablet_manager.cc:1909] T 113a070e074040d29beaf231ef6dbe7c P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:29.379910 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 113a070e074040d29beaf231ef6dbe7c (table rebalancer_dynamic_table_new_name [id=c50eaa08a2ec4b31923ef2396c6b1f9d]) successfully deleted
I20240501 08:12:29.389243 18403 ts_tablet_manager.cc:1888] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:29.389492 18403 log.cc:1196] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/113a070e074040d29beaf231ef6dbe7c
I20240501 08:12:29.390590 18403 ts_tablet_manager.cc:1909] T 113a070e074040d29beaf231ef6dbe7c P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:29.391852 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 113a070e074040d29beaf231ef6dbe7c (table rebalancer_dynamic_table_new_name [id=c50eaa08a2ec4b31923ef2396c6b1f9d]) successfully deleted
I20240501 08:12:29.443001 13344 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 08:12:29.447818 12832 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:29.457724 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 74b3705f2dbf42268e1a2d475b8c811c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a83534623934e398545b34531f3f7f7]), partition=
I20240501 08:12:29.458194 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b3705f2dbf42268e1a2d475b8c811c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.461532 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 74b3705f2dbf42268e1a2d475b8c811c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a83534623934e398545b34531f3f7f7]), partition=
I20240501 08:12:29.461948 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b3705f2dbf42268e1a2d475b8c811c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.463492 16003 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:29.477056 18074 consensus_queue.cc:1035] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.478248 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 74b3705f2dbf42268e1a2d475b8c811c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0a83534623934e398545b34531f3f7f7]), partition=
I20240501 08:12:29.478659 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b3705f2dbf42268e1a2d475b8c811c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.486043 18319 tablet_bootstrap.cc:492] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:29.488147 18319 tablet_bootstrap.cc:654] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.502610 18425 tablet_bootstrap.cc:492] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:29.506279 15870 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:29.509012 13213 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:29.509277 18423 tablet_bootstrap.cc:492] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:29.511404 18339 raft_consensus.cc:482] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.511613 18339 raft_consensus.cc:504] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:29.512446 18339 leader_election.cc:288] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:29.513653 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dccc451745459d8dca19f9ebe2e47f" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:29.513963 13363 raft_consensus.cc:2379] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cc32f6e2d49477f9a23ce6e444039fb in current term 1: Already voted for candidate 5dd722c770d34d6c9154a87917117161 in this term.
I20240501 08:12:29.514998 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0dccc451745459d8dca19f9ebe2e47f" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:29.515390 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:29.516764 18425 tablet_bootstrap.cc:654] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.517935 18423 tablet_bootstrap.cc:654] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.515902 16092 leader_election.cc:302] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4ecb724cb4764a53988fde00407db4c0, 5dd722c770d34d6c9154a87917117161
I20240501 08:12:29.538729 18339 raft_consensus.cc:3037] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.542014 18339 raft_consensus.cc:2726] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:29.553212 18339 consensus_queue.cc:1035] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.515471 18319 tablet_bootstrap.cc:492] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:29.562716 18319 ts_tablet_manager.cc:1367] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.004s
I20240501 08:12:29.564615 18319 raft_consensus.cc:348] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.565029 18319 raft_consensus.cc:374] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.565186 18319 raft_consensus.cc:729] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.565485 18319 consensus_queue.cc:260] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.574631 18319 ts_tablet_manager.cc:1398] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 08:12:29.578406 16137 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:29.580886 18310 consensus_queue.cc:1035] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.608007 18310 consensus_queue.cc:1035] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.671435 18064 consensus_queue.cc:1035] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.685042 12827 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:29.688258 18423 tablet_bootstrap.cc:492] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:29.688453 18423 ts_tablet_manager.cc:1367] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.179s user 0.014s sys 0.043s
I20240501 08:12:29.700868 15870 tablet_service.cc:1467] Processing CreateTablet for tablet b2b0c752d9614f2fbd3863d92df609a4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.701390 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b0c752d9614f2fbd3863d92df609a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.704727 16137 tablet_service.cc:1467] Processing CreateTablet for tablet b2b0c752d9614f2fbd3863d92df609a4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.705174 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b0c752d9614f2fbd3863d92df609a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.709702 13213 tablet_service.cc:1467] Processing CreateTablet for tablet b2b0c752d9614f2fbd3863d92df609a4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.710187 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b0c752d9614f2fbd3863d92df609a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.711625 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 2523847da7464e9f95b18a2e88d813b4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.712095 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2523847da7464e9f95b18a2e88d813b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.713279 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 2523847da7464e9f95b18a2e88d813b4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.713707 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2523847da7464e9f95b18a2e88d813b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.713768 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 2523847da7464e9f95b18a2e88d813b4 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.714187 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2523847da7464e9f95b18a2e88d813b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.716084 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 9102cc8e9b92469ba2ccaa25fea2cc01 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.716833 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 9102cc8e9b92469ba2ccaa25fea2cc01 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.717311 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9102cc8e9b92469ba2ccaa25fea2cc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.718600 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9102cc8e9b92469ba2ccaa25fea2cc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.723582 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 9102cc8e9b92469ba2ccaa25fea2cc01 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:29.724258 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9102cc8e9b92469ba2ccaa25fea2cc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:29.733002 18319 tablet_bootstrap.cc:492] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:29.733492 18423 raft_consensus.cc:348] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.733882 18423 raft_consensus.cc:374] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.734061 18423 raft_consensus.cc:729] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.734694 18319 tablet_bootstrap.cc:654] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.734526 18423 consensus_queue.cc:260] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.738276 18074 raft_consensus.cc:482] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.738641 18074 raft_consensus.cc:504] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.742208 18423 ts_tablet_manager.cc:1398] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.054s user 0.003s sys 0.005s
I20240501 08:12:29.743991 18423 tablet_bootstrap.cc:492] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:29.745998 18423 tablet_bootstrap.cc:654] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.748579 18202 consensus_queue.cc:1035] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:29.756798 18425 tablet_bootstrap.cc:492] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:29.756985 18425 ts_tablet_manager.cc:1367] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.255s user 0.016s sys 0.042s
I20240501 08:12:29.757812 18425 raft_consensus.cc:348] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.758059 18425 raft_consensus.cc:374] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.758169 18425 raft_consensus.cc:729] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.758410 18425 consensus_queue.cc:260] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.759001 18425 ts_tablet_manager.cc:1398] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:29.759315 18425 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:29.762539 18425 tablet_bootstrap.cc:654] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.769683 18307 raft_consensus.cc:482] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.769889 18307 raft_consensus.cc:504] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.770989 18307 leader_election.cc:288] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:29.771054 18438 tablet_bootstrap.cc:492] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:29.773538 18438 tablet_bootstrap.cc:654] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.777477 18423 tablet_bootstrap.cc:492] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:29.778358 18423 ts_tablet_manager.cc:1367] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.009s
I20240501 08:12:29.778985 18425 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:29.779268 18425 ts_tablet_manager.cc:1367] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20240501 08:12:29.779767 18423 raft_consensus.cc:348] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:29.780025 18423 raft_consensus.cc:374] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.780098 18423 raft_consensus.cc:729] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.781438 18423 consensus_queue.cc:260] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:29.782459 18423 ts_tablet_manager.cc:1398] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 08:12:29.783581 18423 tablet_bootstrap.cc:492] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:29.783877 18319 tablet_bootstrap.cc:492] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:29.784044 18425 raft_consensus.cc:348] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:29.785985 18423 tablet_bootstrap.cc:654] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.789665 18425 raft_consensus.cc:374] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.784544 18319 ts_tablet_manager.cc:1367] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.006s
I20240501 08:12:29.790690 18425 raft_consensus.cc:729] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.791185 18425 consensus_queue.cc:260] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:29.794669 18319 raft_consensus.cc:348] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:29.792740 18425 ts_tablet_manager.cc:1398] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240501 08:12:29.797080 18319 raft_consensus.cc:374] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.797665 18319 raft_consensus.cc:729] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.798007 18425 tablet_bootstrap.cc:492] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:29.798480 18319 consensus_queue.cc:260] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:29.805316 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3705f2dbf42268e1a2d475b8c811c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:29.805613 15892 raft_consensus.cc:2454] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:29.806177 18319 ts_tablet_manager.cc:1398] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240501 08:12:29.806564 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3705f2dbf42268e1a2d475b8c811c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:29.806803 16154 raft_consensus.cc:2454] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:29.807138 18438 tablet_bootstrap.cc:492] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:29.808223 18425 tablet_bootstrap.cc:654] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:29.808223 18438 ts_tablet_manager.cc:1367] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20240501 08:12:29.810249 18438 raft_consensus.cc:348] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:29.812393 18438 raft_consensus.cc:374] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.812664 18438 raft_consensus.cc:729] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
W20240501 08:12:29.813514 15823 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 65070.1us
I20240501 08:12:29.814955 18438 consensus_queue.cc:260] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:29.817711 18438 ts_tablet_manager.cc:1398] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240501 08:12:29.821925 18439 raft_consensus.cc:482] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.822103 18439 raft_consensus.cc:504] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:29.822969 18439 leader_election.cc:288] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:29.823761 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2523847da7464e9f95b18a2e88d813b4" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:29.823942 18425 tablet_bootstrap.cc:492] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:29.824061 18425 ts_tablet_manager.cc:1367] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
W20240501 08:12:29.824487 15826 leader_election.cc:341] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:29.824735 18425 raft_consensus.cc:348] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:29.824913 18425 raft_consensus.cc:374] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:29.825003 18425 raft_consensus.cc:729] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:29.826452 18425 consensus_queue.cc:260] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:29.826911 18425 ts_tablet_manager.cc:1398] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:29.840919 17962 raft_consensus.cc:482] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:29.841205 17962 raft_consensus.cc:504] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:29.842396 17962 leader_election.cc:288] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:29.843772 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b0c752d9614f2fbd3863d92df609a4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:29.844954 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b0c752d9614f2fbd3863d92df609a4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:29.845259 16154 raft_consensus.cc:2454] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:29.845546 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:29.845912 13168 leader_election.cc:302] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 6ed448d04e4646658c0de37d829ff15b; no voters:
W20240501 08:12:29.847021 13168 leader_election.cc:341] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:29.847867 17962 raft_consensus.cc:2781] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.848023 17962 raft_consensus.cc:482] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.848136 17962 raft_consensus.cc:3037] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.852023 17962 raft_consensus.cc:504] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:29.854044 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b0c752d9614f2fbd3863d92df609a4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
W20240501 08:12:29.854835 13168 leader_election.cc:341] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:29.856046 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b0c752d9614f2fbd3863d92df609a4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:29.856359 16154 raft_consensus.cc:3037] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.860100 16154 raft_consensus.cc:2454] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:29.862267 13168 leader_election.cc:302] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 6ed448d04e4646658c0de37d829ff15b; no voters: 4d516916eeac468db514eeb928f06190
I20240501 08:12:29.863447 17962 leader_election.cc:288] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:29.863868 17962 raft_consensus.cc:2781] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:29.864246 17962 raft_consensus.cc:686] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:29.864919 17962 consensus_queue.cc:237] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:29.884418 12826 catalog_manager.cc:5484] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
W20240501 08:12:29.905929 15825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 218349us
I20240501 08:12:29.908033 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2523847da7464e9f95b18a2e88d813b4" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:29.908404 15755 raft_consensus.cc:2454] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:29.915166 15825 leader_election.cc:302] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters: 76d830e199eb461c8ca689c9460ee0c2
W20240501 08:12:29.915882 15689 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.6.56.126:44861, user_credentials={real_user=slave}} blocked reactor thread for 95819.9us
I20240501 08:12:29.916115 18443 raft_consensus.cc:2781] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.916210 18443 raft_consensus.cc:482] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.916322 18443 raft_consensus.cc:3037] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.917362 15689 leader_election.cc:302] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:29.917618 18307 raft_consensus.cc:2781] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.917716 18307 raft_consensus.cc:482] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.917817 18307 raft_consensus.cc:3037] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.919231 18443 raft_consensus.cc:504] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:29.919909 18443 leader_election.cc:288] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:29.920701 18307 raft_consensus.cc:504] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.921597 18307 leader_election.cc:288] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:29.923527 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2523847da7464e9f95b18a2e88d813b4" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:29.924424 15755 raft_consensus.cc:3037] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.928131 18074 leader_election.cc:288] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:29.929728 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3705f2dbf42268e1a2d475b8c811c" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:29.930042 15893 raft_consensus.cc:2454] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:29.930387 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2523847da7464e9f95b18a2e88d813b4" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:29.930979 16090 leader_election.cc:302] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:29.931455 15755 raft_consensus.cc:2454] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
W20240501 08:12:29.931972 15826 leader_election.cc:341] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:29.932365 18074 raft_consensus.cc:2781] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:29.932507 18074 raft_consensus.cc:482] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:29.932632 18074 raft_consensus.cc:3037] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.936877 18074 raft_consensus.cc:504] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.938556 15825 leader_election.cc:302] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:29.939019 18436 raft_consensus.cc:2781] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:29.939162 18436 raft_consensus.cc:686] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:29.939455 18436 consensus_queue.cc:237] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:29.942090 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3705f2dbf42268e1a2d475b8c811c" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:29.942382 15892 raft_consensus.cc:3037] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:29.946481 15892 raft_consensus.cc:2454] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:29.947064 16090 leader_election.cc:302] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:29.947480 18339 raft_consensus.cc:2781] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:29.947641 18339 raft_consensus.cc:686] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:29.947921 18339 consensus_queue.cc:237] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:29.951150 18074 leader_election.cc:288] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:29.957741 12826 catalog_manager.cc:5484] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:29.960685 12832 catalog_manager.cc:5484] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:29.961596 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3705f2dbf42268e1a2d475b8c811c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:29.961961 15892 raft_consensus.cc:2379] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 982f49fc87f149439cda0de2261b6bfd in current term 1: Already voted for candidate 6cc32f6e2d49477f9a23ce6e444039fb in this term.
I20240501 08:12:29.963461 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3705f2dbf42268e1a2d475b8c811c" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:29.974236 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3705f2dbf42268e1a2d475b8c811c" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:29.974709 15754 raft_consensus.cc:2379] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cc32f6e2d49477f9a23ce6e444039fb in current term 1: Already voted for candidate 982f49fc87f149439cda0de2261b6bfd in this term.
I20240501 08:12:29.975301 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3705f2dbf42268e1a2d475b8c811c" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:29.975564 15754 raft_consensus.cc:2379] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6cc32f6e2d49477f9a23ce6e444039fb in current term 1: Already voted for candidate 982f49fc87f149439cda0de2261b6bfd in this term.
I20240501 08:12:29.976820 15689 leader_election.cc:302] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:29.977733 18217 raft_consensus.cc:2726] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:12:30.008711 18423 tablet_bootstrap.cc:492] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:30.008903 18423 ts_tablet_manager.cc:1367] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.225s user 0.026s sys 0.053s
I20240501 08:12:30.009769 18423 raft_consensus.cc:348] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:30.010001 18423 raft_consensus.cc:374] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.010102 18423 raft_consensus.cc:729] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.010332 18423 consensus_queue.cc:260] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:30.010926 15735 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:30.012022 18423 ts_tablet_manager.cc:1398] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:30.012462 18423 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:30.016685 18423 tablet_bootstrap.cc:654] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.029016 18437 tablet_bootstrap.cc:492] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:30.029532 18307 raft_consensus.cc:482] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.029767 18307 raft_consensus.cc:504] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:30.033651 18307 leader_election.cc:288] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:30.037927 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102cc8e9b92469ba2ccaa25fea2cc01" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
W20240501 08:12:30.040448 15692 leader_election.cc:341] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:30.032904 18437 tablet_bootstrap.cc:654] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.044246 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102cc8e9b92469ba2ccaa25fea2cc01" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
W20240501 08:12:30.045769 15689 leader_election.cc:341] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:12:30.046963 13041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 49908.8us
I20240501 08:12:30.060179 18437 tablet_bootstrap.cc:492] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:30.060555 18437 ts_tablet_manager.cc:1367] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.006s
I20240501 08:12:30.062503 18437 raft_consensus.cc:348] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:30.063012 18437 raft_consensus.cc:374] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.063189 18437 raft_consensus.cc:729] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.063647 18437 consensus_queue.cc:260] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:30.066619 18437 ts_tablet_manager.cc:1398] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:12:30.067030 18437 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:30.045924 15689 leader_election.cc:302] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:30.068729 18217 raft_consensus.cc:2726] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:30.077934 18437 tablet_bootstrap.cc:654] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.091648 18437 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:30.091858 18437 ts_tablet_manager.cc:1367] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.006s
I20240501 08:12:30.092689 18437 raft_consensus.cc:348] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:30.092945 18437 raft_consensus.cc:374] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.093067 18437 raft_consensus.cc:729] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.093334 18437 consensus_queue.cc:260] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:30.095355 18437 ts_tablet_manager.cc:1398] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:12:30.096199 18423 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:30.096398 18423 ts_tablet_manager.cc:1367] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.084s user 0.020s sys 0.038s
I20240501 08:12:30.097278 18423 raft_consensus.cc:348] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:30.097514 18423 raft_consensus.cc:374] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.097601 18423 raft_consensus.cc:729] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.097856 18423 consensus_queue.cc:260] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:30.098538 18423 ts_tablet_manager.cc:1398] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:30.103737 17701 raft_consensus.cc:482] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.103940 17701 raft_consensus.cc:504] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:30.104686 17701 leader_election.cc:288] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:30.105531 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2523847da7464e9f95b18a2e88d813b4" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:30.106115 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2523847da7464e9f95b18a2e88d813b4" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:30.106397 15756 raft_consensus.cc:2379] T 2523847da7464e9f95b18a2e88d813b4 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76d830e199eb461c8ca689c9460ee0c2 in current term 1: Already voted for candidate 4d516916eeac468db514eeb928f06190 in this term.
I20240501 08:12:30.106897 12909 leader_election.cc:302] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2; no voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:30.107331 17701 raft_consensus.cc:3037] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.113299 17701 raft_consensus.cc:2726] T 2523847da7464e9f95b18a2e88d813b4 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:30.164186 12832 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:30.174125 15892 raft_consensus.cc:1261] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:30.174124 15756 raft_consensus.cc:1261] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:30.174923 18074 consensus_queue.cc:1035] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.176131 18074 consensus_queue.cc:1035] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.185986 12832 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 74b3705f2dbf42268e1a2d475b8c811c (table rebalancer_dynamic_table [id=0a83534623934e398545b34531f3f7f7]). Expected version 1 got 0
I20240501 08:12:30.196807 18436 raft_consensus.cc:482] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.197095 18436 raft_consensus.cc:504] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:30.198046 18436 leader_election.cc:288] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:30.199752 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102cc8e9b92469ba2ccaa25fea2cc01" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:30.200145 12974 raft_consensus.cc:2454] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:30.202188 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102cc8e9b92469ba2ccaa25fea2cc01" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:30.202436 15755 raft_consensus.cc:2454] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:30.204517 15826 leader_election.cc:302] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:30.205214 18439 raft_consensus.cc:2781] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:30.205358 18439 raft_consensus.cc:482] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:30.205457 18439 raft_consensus.cc:3037] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.210916 18468 tablet.cc:1722] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:30.211849 18468 tablet.cc:1620] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:30.221092 12826 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 74b3705f2dbf42268e1a2d475b8c811c (table rebalancer_dynamic_table [id=0a83534623934e398545b34531f3f7f7]). Expected version 1 got 0
I20240501 08:12:30.230312 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=0a83534623934e398545b34531f3f7f7] alter complete (version 1)
I20240501 08:12:30.232713 18439 raft_consensus.cc:504] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:30.235699 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102cc8e9b92469ba2ccaa25fea2cc01" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:30.235953 15755 raft_consensus.cc:3037] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.239173 15755 raft_consensus.cc:2454] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:30.243240 15825 leader_election.cc:302] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:30.243538 18443 raft_consensus.cc:2781] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:30.243680 18443 raft_consensus.cc:686] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:30.243965 18443 consensus_queue.cc:237] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:30.244657 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9102cc8e9b92469ba2ccaa25fea2cc01" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:30.244889 12974 raft_consensus.cc:3037] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.251011 12974 raft_consensus.cc:2454] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:30.251389 18439 leader_election.cc:288] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:30.260715 18471 tablet.cc:1722] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:30.265599 18451 consensus_queue.cc:1035] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.278332 12832 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 74b3705f2dbf42268e1a2d475b8c811c (table rebalancer_dynamic_table [id=0a83534623934e398545b34531f3f7f7]). Expected version 1 got 0
I20240501 08:12:30.279991 18471 tablet.cc:1620] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:30.280552 18473 tablet.cc:1722] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:30.281682 18473 tablet.cc:1620] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:30.281942 18439 raft_consensus.cc:482] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:30.282096 18439 raft_consensus.cc:504] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:30.283063 18439 leader_election.cc:288] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:30.283077 12832 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 74b3705f2dbf42268e1a2d475b8c811c (table rebalancer_dynamic_table [id=0a83534623934e398545b34531f3f7f7]). Expected version 1 got 0
I20240501 08:12:30.284137 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b0c752d9614f2fbd3863d92df609a4" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:30.286303 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2b0c752d9614f2fbd3863d92df609a4" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:30.290441 15823 leader_election.cc:302] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 6cc32f6e2d49477f9a23ce6e444039fb, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:30.292194 18436 raft_consensus.cc:2726] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:30.294122 12826 catalog_manager.cc:5484] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:30.295423 15892 raft_consensus.cc:3037] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:30.301218 18451 consensus_queue.cc:1035] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.321036 16025 consensus_queue.cc:237] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:30.323381 13235 raft_consensus.cc:1261] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:30.323760 15756 raft_consensus.cc:1261] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:30.323990 18345 consensus_queue.cc:1035] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:30.324354 18202 consensus_queue.cc:1035] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:30.327636 18345 raft_consensus.cc:2932] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:30.328366 15756 raft_consensus.cc:2932] T 8a48ba4666c14848b2b5a1e62f16357b P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:30.334290 12826 catalog_manager.cc:5484] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:30.337070 12829 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:30.339215 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a48ba4666c14848b2b5a1e62f16357b with delete_type TABLET_DATA_TOMBSTONED (TS e3fefd4dca2d41678187047f06c10187 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:30.340394 18443 consensus_queue.cc:1035] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.348264 18493 tablet_replica.cc:330] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:30.348644 18493 raft_consensus.cc:2229] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:30.349079 18493 raft_consensus.cc:2258] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:30.350121 18493 ts_tablet_manager.cc:1875] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:30.353396 18468 tablet.cc:1722] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:30.353897 18468 tablet.cc:1620] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:30.354796 18471 tablet.cc:1722] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:30.355232 18471 tablet.cc:1620] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:30.356995 18473 tablet.cc:1722] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:30.357455 18473 tablet.cc:1620] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:30.364864 12829 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=0a83534623934e398545b34531f3f7f7] alter complete (version 2)
I20240501 08:12:30.365504 18493 ts_tablet_manager.cc:1888] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:30.365751 18493 log.cc:1196] T 8a48ba4666c14848b2b5a1e62f16357b P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/8a48ba4666c14848b2b5a1e62f16357b
I20240501 08:12:30.366932 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 8a48ba4666c14848b2b5a1e62f16357b (table rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]) successfully deleted
I20240501 08:12:30.371198 18436 consensus_queue.cc:1035] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.373749 13235 raft_consensus.cc:2932] T 8a48ba4666c14848b2b5a1e62f16357b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:30.467533 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:30.481871 18468 tablet.cc:1722] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:30.483371 18471 tablet.cc:1722] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:30.496675 18473 tablet.cc:1722] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:30.509385 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=0a83534623934e398545b34531f3f7f7] alter complete (version 3)
I20240501 08:12:30.611860 12826 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:30.612103 12826 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:30.615305 12826 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 74b3705f2dbf42268e1a2d475b8c811c
I20240501 08:12:30.616369 16137 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b3705f2dbf42268e1a2d475b8c811c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:30 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:30.616382 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b3705f2dbf42268e1a2d475b8c811c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:30 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:30.616677 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b3705f2dbf42268e1a2d475b8c811c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:30 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:30.626253 18443 consensus_queue.cc:1035] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.634908 18544 tablet_replica.cc:330] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:30.635567 18542 tablet_replica.cc:330] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:30.635866 18542 raft_consensus.cc:2229] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:30.636143 18542 raft_consensus.cc:2258] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:30.638392 18542 ts_tablet_manager.cc:1875] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:30.636175 18544 raft_consensus.cc:2229] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:30.640002 18544 raft_consensus.cc:2258] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:30.639923 18439 consensus_queue.cc:1035] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:30.640622 18543 tablet_replica.cc:330] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:30.640848 18543 raft_consensus.cc:2229] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:30.645913 18544 ts_tablet_manager.cc:1875] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:30.647053 18543 raft_consensus.cc:2258] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:30.648020 18543 ts_tablet_manager.cc:1875] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:30.651237 18542 ts_tablet_manager.cc:1888] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:30.651407 18542 log.cc:1196] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/74b3705f2dbf42268e1a2d475b8c811c
I20240501 08:12:30.651923 18542 ts_tablet_manager.cc:1909] T 74b3705f2dbf42268e1a2d475b8c811c P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:30.653144 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 74b3705f2dbf42268e1a2d475b8c811c (table rebalancer_dynamic_table_new_name [id=0a83534623934e398545b34531f3f7f7]) successfully deleted
I20240501 08:12:30.654882 18543 ts_tablet_manager.cc:1888] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:30.655014 18543 log.cc:1196] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/74b3705f2dbf42268e1a2d475b8c811c
I20240501 08:12:30.655325 18544 ts_tablet_manager.cc:1888] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:30.655462 18544 log.cc:1196] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/74b3705f2dbf42268e1a2d475b8c811c
I20240501 08:12:30.655475 18543 ts_tablet_manager.cc:1909] T 74b3705f2dbf42268e1a2d475b8c811c P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:30.655982 18544 ts_tablet_manager.cc:1909] T 74b3705f2dbf42268e1a2d475b8c811c P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:30.657243 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 74b3705f2dbf42268e1a2d475b8c811c (table rebalancer_dynamic_table_new_name [id=0a83534623934e398545b34531f3f7f7]) successfully deleted
I20240501 08:12:30.660945 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 74b3705f2dbf42268e1a2d475b8c811c (table rebalancer_dynamic_table_new_name [id=0a83534623934e398545b34531f3f7f7]) successfully deleted
I20240501 08:12:30.720191 12827 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:30.729537 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 76fdcb09c5d242058b9954832855f5cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53ea9064c91d47dbb10c13383f524daa]), partition=
I20240501 08:12:30.729658 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 76fdcb09c5d242058b9954832855f5cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53ea9064c91d47dbb10c13383f524daa]), partition=
I20240501 08:12:30.729925 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fdcb09c5d242058b9954832855f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.730082 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fdcb09c5d242058b9954832855f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.733445 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 76fdcb09c5d242058b9954832855f5cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53ea9064c91d47dbb10c13383f524daa]), partition=
I20240501 08:12:30.733896 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fdcb09c5d242058b9954832855f5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.751228 18568 tablet_bootstrap.cc:492] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:30.753875 18568 tablet_bootstrap.cc:654] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.757551 18569 tablet_bootstrap.cc:492] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:30.759249 18569 tablet_bootstrap.cc:654] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.760344 18566 tablet_bootstrap.cc:492] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:30.768669 18566 tablet_bootstrap.cc:654] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.775632 18568 tablet_bootstrap.cc:492] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:30.776055 18568 ts_tablet_manager.cc:1367] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.003s
I20240501 08:12:30.777254 18568 raft_consensus.cc:348] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.777879 18568 raft_consensus.cc:374] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.778064 18568 raft_consensus.cc:729] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.778460 18568 consensus_queue.cc:260] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.781033 18568 ts_tablet_manager.cc:1398] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:12:30.799027 18569 tablet_bootstrap.cc:492] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:30.799209 18569 ts_tablet_manager.cc:1367] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.006s
I20240501 08:12:30.800016 18569 raft_consensus.cc:348] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.800246 18569 raft_consensus.cc:374] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.800374 18569 raft_consensus.cc:729] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.800643 18569 consensus_queue.cc:260] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.801419 18569 ts_tablet_manager.cc:1398] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:30.827399 18566 tablet_bootstrap.cc:492] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:30.827627 18566 ts_tablet_manager.cc:1367] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.031s
I20240501 08:12:30.828979 18566 raft_consensus.cc:348] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.829188 18566 raft_consensus.cc:374] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.829272 18566 raft_consensus.cc:729] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.829545 18566 consensus_queue.cc:260] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.831395 18566 ts_tablet_manager.cc:1398] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:30.847435 12827 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:30.862545 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2794da46e543b8a25a8e65b229f95e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.863121 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2794da46e543b8a25a8e65b229f95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.869429 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2794da46e543b8a25a8e65b229f95e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.869947 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2794da46e543b8a25a8e65b229f95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.872741 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2794da46e543b8a25a8e65b229f95e (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.872941 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 8c487b2c7fe24515a8970a07883c126f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.873245 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2794da46e543b8a25a8e65b229f95e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.873430 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c487b2c7fe24515a8970a07883c126f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.874984 15870 tablet_service.cc:1467] Processing CreateTablet for tablet a6154bf2fadb45e58c27c46165b5824f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.875387 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6154bf2fadb45e58c27c46165b5824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.878072 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 8c487b2c7fe24515a8970a07883c126f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.878494 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c487b2c7fe24515a8970a07883c126f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.880683 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 8c487b2c7fe24515a8970a07883c126f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.881091 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c487b2c7fe24515a8970a07883c126f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.882553 13344 tablet_service.cc:1467] Processing CreateTablet for tablet a6154bf2fadb45e58c27c46165b5824f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.882995 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6154bf2fadb45e58c27c46165b5824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.890481 16138 tablet_service.cc:1467] Processing CreateTablet for tablet a6154bf2fadb45e58c27c46165b5824f (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:30.891044 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6154bf2fadb45e58c27c46165b5824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:30.894799 18568 tablet_bootstrap.cc:492] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:30.897126 18568 tablet_bootstrap.cc:654] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.903280 18587 tablet_bootstrap.cc:492] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:30.904796 18588 tablet_bootstrap.cc:492] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:30.906183 18566 tablet_bootstrap.cc:492] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:30.906777 18588 tablet_bootstrap.cc:654] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.907804 18566 tablet_bootstrap.cc:654] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.912974 18587 tablet_bootstrap.cc:654] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.914533 18568 tablet_bootstrap.cc:492] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:30.914772 18568 ts_tablet_manager.cc:1367] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20240501 08:12:30.915848 18568 raft_consensus.cc:348] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.916539 18568 raft_consensus.cc:374] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.918725 18568 raft_consensus.cc:729] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.919231 18568 consensus_queue.cc:260] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.920039 18568 ts_tablet_manager.cc:1398] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:12:30.921057 18568 tablet_bootstrap.cc:492] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:30.922833 18568 tablet_bootstrap.cc:654] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.927897 18568 tablet_bootstrap.cc:492] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:30.928061 18568 ts_tablet_manager.cc:1367] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 08:12:30.928186 18566 tablet_bootstrap.cc:492] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:30.928419 18566 ts_tablet_manager.cc:1367] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20240501 08:12:30.928897 18568 raft_consensus.cc:348] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.929121 18568 raft_consensus.cc:374] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.929230 18568 raft_consensus.cc:729] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.929451 18568 consensus_queue.cc:260] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.930202 18568 ts_tablet_manager.cc:1398] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:30.929550 18566 raft_consensus.cc:348] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:30.934511 18566 raft_consensus.cc:374] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.935070 18566 raft_consensus.cc:729] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.935637 18566 consensus_queue.cc:260] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:30.937757 18566 ts_tablet_manager.cc:1398] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:30.938158 18566 tablet_bootstrap.cc:492] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:30.941148 18566 tablet_bootstrap.cc:654] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.949321 18588 tablet_bootstrap.cc:492] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:30.949296 18592 tablet_bootstrap.cc:492] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:30.949594 18588 ts_tablet_manager.cc:1367] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.000s
I20240501 08:12:30.950645 18588 raft_consensus.cc:348] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.951085 18588 raft_consensus.cc:374] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.951332 18592 tablet_bootstrap.cc:654] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:30.954075 18588 raft_consensus.cc:729] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.954393 18588 consensus_queue.cc:260] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.963927 18588 ts_tablet_manager.cc:1398] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240501 08:12:30.968252 15870 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:12:30.989854 13344 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 08:12:30.996366 18587 tablet_bootstrap.cc:492] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:30.996727 18587 ts_tablet_manager.cc:1367] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.094s user 0.000s sys 0.006s
I20240501 08:12:30.997771 18587 raft_consensus.cc:348] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:30.998075 18587 raft_consensus.cc:374] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:30.998193 18587 raft_consensus.cc:729] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:30.998490 18587 consensus_queue.cc:260] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.003012 18587 ts_tablet_manager.cc:1398] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 08:12:31.006839 18587 tablet_bootstrap.cc:492] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:31.009404 18587 tablet_bootstrap.cc:654] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.023619 18592 tablet_bootstrap.cc:492] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:31.023885 18592 ts_tablet_manager.cc:1367] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.075s user 0.014s sys 0.014s
I20240501 08:12:31.025817 18592 raft_consensus.cc:348] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.026163 18592 raft_consensus.cc:374] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.026297 18592 raft_consensus.cc:729] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.026572 18592 consensus_queue.cc:260] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.028492 18592 ts_tablet_manager.cc:1398] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:31.029928 18592 tablet_bootstrap.cc:492] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:31.038420 18592 tablet_bootstrap.cc:654] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.043934 18587 tablet_bootstrap.cc:492] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:31.044103 18461 raft_consensus.cc:482] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.044160 18587 ts_tablet_manager.cc:1367] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.001s
I20240501 08:12:31.044340 18461 raft_consensus.cc:504] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.045020 18461 leader_election.cc:288] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:31.045099 18587 raft_consensus.cc:348] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.045429 18587 raft_consensus.cc:374] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.045567 18587 raft_consensus.cc:729] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.045917 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6154bf2fadb45e58c27c46165b5824f" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:31.045945 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6154bf2fadb45e58c27c46165b5824f" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:31.046149 15892 raft_consensus.cc:2454] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:31.046162 16154 raft_consensus.cc:2454] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:31.046583 13298 leader_election.cc:302] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters:
I20240501 08:12:31.046880 18587 consensus_queue.cc:260] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.047070 18461 raft_consensus.cc:2781] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:31.047201 18461 raft_consensus.cc:482] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:31.047308 18461 raft_consensus.cc:3037] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.048034 18592 tablet_bootstrap.cc:492] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:31.048228 18592 ts_tablet_manager.cc:1367] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
I20240501 08:12:31.048982 18587 ts_tablet_manager.cc:1398] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:31.050863 18461 raft_consensus.cc:504] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.051849 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6154bf2fadb45e58c27c46165b5824f" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:31.052055 15892 raft_consensus.cc:3037] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.055027 15892 raft_consensus.cc:2454] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:31.055392 18592 raft_consensus.cc:348] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.055626 18592 raft_consensus.cc:374] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.055598 13298 leader_election.cc:302] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters:
I20240501 08:12:31.055711 18592 raft_consensus.cc:729] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.055964 18310 raft_consensus.cc:2781] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:31.055999 18592 consensus_queue.cc:260] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.065779 18592 ts_tablet_manager.cc:1398] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 08:12:31.073280 13213 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:12:31.076457 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6154bf2fadb45e58c27c46165b5824f" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:31.076736 16154 raft_consensus.cc:3037] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.079759 16154 raft_consensus.cc:2454] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:31.079911 16138 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:31.081293 18461 leader_election.cc:288] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:31.088562 18443 raft_consensus.cc:482] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.088737 18443 raft_consensus.cc:504] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.089563 18443 leader_election.cc:288] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:31.090140 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c487b2c7fe24515a8970a07883c126f" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:31.090230 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c487b2c7fe24515a8970a07883c126f" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:31.090468 15756 raft_consensus.cc:2454] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:31.090495 13235 raft_consensus.cc:2454] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:31.090953 15825 leader_election.cc:302] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:31.091380 18443 raft_consensus.cc:2781] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:31.091487 18443 raft_consensus.cc:482] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:31.091562 18443 raft_consensus.cc:3037] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.102388 18443 raft_consensus.cc:504] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.104121 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c487b2c7fe24515a8970a07883c126f" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:31.104405 15756 raft_consensus.cc:3037] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.110275 15756 raft_consensus.cc:2454] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:31.112715 15825 leader_election.cc:302] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:31.113207 18439 raft_consensus.cc:2781] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:31.113333 18439 raft_consensus.cc:686] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:31.113598 18439 consensus_queue.cc:237] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.116585 18310 raft_consensus.cc:686] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:31.117630 18310 consensus_queue.cc:237] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.128391 18443 leader_election.cc:288] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:31.128840 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c487b2c7fe24515a8970a07883c126f" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:31.129117 13235 raft_consensus.cc:3037] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.130421 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:31.130808 18339 raft_consensus.cc:482] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.131014 18339 raft_consensus.cc:504] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.132122 18339 leader_election.cc:288] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:31.132062 13235 raft_consensus.cc:2454] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:31.134611 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fdcb09c5d242058b9954832855f5cd" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:31.134897 12974 raft_consensus.cc:2454] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:31.135465 16093 leader_election.cc:302] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:31.135797 18339 raft_consensus.cc:2781] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:31.135877 18339 raft_consensus.cc:482] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:31.135968 18339 raft_consensus.cc:3037] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.138343 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fdcb09c5d242058b9954832855f5cd" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:31.138720 18339 raft_consensus.cc:504] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.138619 15756 raft_consensus.cc:2454] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:31.139952 18449 raft_consensus.cc:482] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.140112 18449 raft_consensus.cc:504] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.141049 18449 leader_election.cc:288] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:31.141949 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2794da46e543b8a25a8e65b229f95e" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:31.142005 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fdcb09c5d242058b9954832855f5cd" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:31.142203 12974 raft_consensus.cc:3037] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:12:31.142555 13170 leader_election.cc:341] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:31.143450 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2794da46e543b8a25a8e65b229f95e" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:31.143745 16154 raft_consensus.cc:2454] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:31.144325 13168 leader_election.cc:302] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 6ed448d04e4646658c0de37d829ff15b; no voters: 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:31.144789 18449 raft_consensus.cc:2781] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:31.144920 18449 raft_consensus.cc:482] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:31.145015 18449 raft_consensus.cc:3037] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.145330 12974 raft_consensus.cc:2454] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:31.145972 16093 leader_election.cc:302] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:31.146302 18430 raft_consensus.cc:2781] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:31.149631 18339 leader_election.cc:288] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:31.151140 18449 raft_consensus.cc:504] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.151566 18430 raft_consensus.cc:686] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:31.152007 18430 consensus_queue.cc:237] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.153358 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2794da46e543b8a25a8e65b229f95e" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:31.153858 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fdcb09c5d242058b9954832855f5cd" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
W20240501 08:12:31.153892 13170 leader_election.cc:341] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:31.154071 15755 raft_consensus.cc:3037] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.154587 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2794da46e543b8a25a8e65b229f95e" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:31.154772 16157 raft_consensus.cc:3037] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.157172 15755 raft_consensus.cc:2454] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:31.158123 16157 raft_consensus.cc:2454] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:31.158676 13168 leader_election.cc:302] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 6ed448d04e4646658c0de37d829ff15b; no voters: 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:31.159035 18451 raft_consensus.cc:2781] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:31.163630 12826 catalog_manager.cc:5484] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:31.165333 18449 leader_election.cc:288] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:31.168541 18451 raft_consensus.cc:686] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:31.169008 18451 consensus_queue.cc:237] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.180826 18566 tablet_bootstrap.cc:492] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:31.181661 12954 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:31.181811 18566 ts_tablet_manager.cc:1367] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.244s user 0.003s sys 0.004s
I20240501 08:12:31.182814 18566 raft_consensus.cc:348] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.183070 18566 raft_consensus.cc:374] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.183167 18566 raft_consensus.cc:729] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.189040 15735 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:31.190770 12826 catalog_manager.cc:5484] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:31.183440 18566 consensus_queue.cc:260] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.211505 18566 ts_tablet_manager.cc:1398] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240501 08:12:31.213783 12830 catalog_manager.cc:5484] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:31.228722 16003 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:31.270980 12826 catalog_manager.cc:5484] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:31.367640 13363 consensus_queue.cc:237] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:31.369727 15755 raft_consensus.cc:1261] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:31.369727 13235 raft_consensus.cc:1261] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:31.376638 18461 consensus_queue.cc:1035] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:31.377480 18310 consensus_queue.cc:1035] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:31.383214 18310 raft_consensus.cc:2932] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:31.383955 13235 raft_consensus.cc:2932] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:31.385984 15755 raft_consensus.cc:2932] T 74d6cce81d164722afad1935d4695374 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:31.389453 12826 catalog_manager.cc:5484] T 74d6cce81d164722afad1935d4695374 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:31.394007 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d6cce81d164722afad1935d4695374 with delete_type TABLET_DATA_TOMBSTONED (TS e3fefd4dca2d41678187047f06c10187 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:31.407832 18641 tablet_replica.cc:330] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:31.408051 18641 raft_consensus.cc:2229] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:31.408326 18641 raft_consensus.cc:2258] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:31.408991 18641 ts_tablet_manager.cc:1875] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:31.414310 18641 ts_tablet_manager.cc:1888] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:31.414541 18641 log.cc:1196] T 74d6cce81d164722afad1935d4695374 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/74d6cce81d164722afad1935d4695374
I20240501 08:12:31.415194 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 74d6cce81d164722afad1935d4695374 (table rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]) successfully deleted
I20240501 08:12:31.459510 18217 raft_consensus.cc:482] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.459758 18217 raft_consensus.cc:504] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.460721 18217 leader_election.cc:288] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:31.461143 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2794da46e543b8a25a8e65b229f95e" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:31.461418 16154 raft_consensus.cc:2379] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 982f49fc87f149439cda0de2261b6bfd in current term 1: Already voted for candidate 6ed448d04e4646658c0de37d829ff15b in this term.
I20240501 08:12:31.461645 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2794da46e543b8a25a8e65b229f95e" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:31.462495 15692 leader_election.cc:302] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 6cc32f6e2d49477f9a23ce6e444039fb, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:31.462852 18217 raft_consensus.cc:3037] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:31.465906 18217 raft_consensus.cc:2726] T 4e2794da46e543b8a25a8e65b229f95e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:31.490550 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:31.499943 18439 consensus_queue.cc:1035] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.500113 18310 consensus_queue.cc:1035] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.517832 15755 raft_consensus.cc:1261] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:31.518870 18430 consensus_queue.cc:1035] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.527165 12974 raft_consensus.cc:1261] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:31.527812 18430 consensus_queue.cc:1035] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.543849 12831 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 76fdcb09c5d242058b9954832855f5cd (table rebalancer_dynamic_table [id=53ea9064c91d47dbb10c13383f524daa]). Expected version 1 got 0
I20240501 08:12:31.552386 12831 catalog_manager.cc:5576] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839) does not have the latest schema for tablet 76fdcb09c5d242058b9954832855f5cd (table rebalancer_dynamic_table [id=53ea9064c91d47dbb10c13383f524daa]). Expected version 1 got 0
I20240501 08:12:31.553540 18677 tablet.cc:1722] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:31.554786 18679 tablet.cc:1722] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:31.555781 18680 tablet.cc:1722] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:31.556732 18680 tablet.cc:1620] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:31.559229 18679 tablet.cc:1620] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:31.562721 12827 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 76fdcb09c5d242058b9954832855f5cd (table rebalancer_dynamic_table [id=53ea9064c91d47dbb10c13383f524daa]). Expected version 1 got 0
I20240501 08:12:31.562809 18677 tablet.cc:1620] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:31.563678 12831 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=53ea9064c91d47dbb10c13383f524daa] alter complete (version 1)
I20240501 08:12:31.630918 18610 consensus_queue.cc:1035] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.652971 18436 consensus_queue.cc:1035] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.669696 18451 consensus_queue.cc:1035] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.680461 12831 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:31.687083 18619 consensus_queue.cc:1035] T 4e2794da46e543b8a25a8e65b229f95e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:31.696949 18677 tablet.cc:1722] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:31.697659 18677 tablet.cc:1620] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:31.698262 18679 tablet.cc:1722] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:31.698873 18679 tablet.cc:1620] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:31.701061 18680 tablet.cc:1722] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:31.701635 18680 tablet.cc:1620] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:31.725698 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=53ea9064c91d47dbb10c13383f524daa] alter complete (version 2)
I20240501 08:12:31.746346 12831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:31.761801 16003 tablet_service.cc:1467] Processing CreateTablet for tablet d241e087e754485cb80080754d27aef7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.762351 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d241e087e754485cb80080754d27aef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.762935 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 321bf6596ff24ec7af79e0510eadedce (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.763394 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321bf6596ff24ec7af79e0510eadedce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.766669 12954 tablet_service.cc:1467] Processing CreateTablet for tablet d241e087e754485cb80080754d27aef7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.766822 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 321bf6596ff24ec7af79e0510eadedce (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.767102 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d241e087e754485cb80080754d27aef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.762256 16004 tablet_service.cc:1467] Processing CreateTablet for tablet 321bf6596ff24ec7af79e0510eadedce (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.767728 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321bf6596ff24ec7af79e0510eadedce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.772938 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 37b0020baddc4f47b625edc2e1180e51 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.773411 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b0020baddc4f47b625edc2e1180e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.774008 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321bf6596ff24ec7af79e0510eadedce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.785455 18701 tablet_bootstrap.cc:492] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:31.789453 18701 tablet_bootstrap.cc:654] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.793474 15870 tablet_service.cc:1467] Processing CreateTablet for tablet d241e087e754485cb80080754d27aef7 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.794107 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d241e087e754485cb80080754d27aef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.794034 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 37b0020baddc4f47b625edc2e1180e51 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.794595 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b0020baddc4f47b625edc2e1180e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.793588 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 37b0020baddc4f47b625edc2e1180e51 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:31.797410 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b0020baddc4f47b625edc2e1180e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:31.827055 18704 tablet_bootstrap.cc:492] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:31.827397 12831 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:31.831818 18702 tablet_bootstrap.cc:492] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:31.832365 18704 tablet_bootstrap.cc:654] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.833475 18706 tablet_bootstrap.cc:492] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:31.837064 18706 tablet_bootstrap.cc:654] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.837208 18702 tablet_bootstrap.cc:654] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.839663 18677 tablet.cc:1722] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:31.842087 18679 tablet.cc:1722] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:31.842837 18680 tablet.cc:1722] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:31.848129 18701 tablet_bootstrap.cc:492] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:31.848309 18701 ts_tablet_manager.cc:1367] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.063s user 0.013s sys 0.023s
I20240501 08:12:31.849144 18701 raft_consensus.cc:348] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:31.849359 18701 raft_consensus.cc:374] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.849442 18701 raft_consensus.cc:729] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.849783 18701 consensus_queue.cc:260] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:31.850375 18701 ts_tablet_manager.cc:1398] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:12:31.852119 12831 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=53ea9064c91d47dbb10c13383f524daa] alter complete (version 3)
I20240501 08:12:31.856472 18704 tablet_bootstrap.cc:492] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:31.856622 18704 ts_tablet_manager.cc:1367] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.002s
I20240501 08:12:31.857235 18704 raft_consensus.cc:348] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.857434 18704 raft_consensus.cc:374] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.857548 18704 raft_consensus.cc:729] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.858186 18704 consensus_queue.cc:260] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.858879 18704 ts_tablet_manager.cc:1398] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:31.859381 18704 tablet_bootstrap.cc:492] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:31.871532 18705 tablet_bootstrap.cc:492] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:31.871584 18704 tablet_bootstrap.cc:654] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.876736 18705 tablet_bootstrap.cc:654] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.879638 18706 tablet_bootstrap.cc:492] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:31.880124 18704 tablet_bootstrap.cc:492] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:31.880115 18702 tablet_bootstrap.cc:492] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:31.880272 18702 ts_tablet_manager.cc:1367] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.004s
I20240501 08:12:31.880378 18704 ts_tablet_manager.cc:1367] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.001s
I20240501 08:12:31.880559 18706 ts_tablet_manager.cc:1367] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.013s
I20240501 08:12:31.881302 18702 raft_consensus.cc:348] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.881592 18702 raft_consensus.cc:374] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.881652 18702 raft_consensus.cc:729] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.881558 18704 raft_consensus.cc:348] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.881753 18704 raft_consensus.cc:374] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.882228 18704 raft_consensus.cc:729] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.881911 18702 consensus_queue.cc:260] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.882546 18704 consensus_queue.cc:260] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.882820 18706 raft_consensus.cc:348] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.883034 18702 ts_tablet_manager.cc:1398] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:31.883236 18704 ts_tablet_manager.cc:1398] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:31.883306 18706 raft_consensus.cc:374] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.883550 18706 raft_consensus.cc:729] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.883962 18706 consensus_queue.cc:260] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.886052 18706 ts_tablet_manager.cc:1398] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:31.892895 18702 tablet_bootstrap.cc:492] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:31.897732 18702 tablet_bootstrap.cc:654] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.910027 18705 tablet_bootstrap.cc:492] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:31.910321 18705 ts_tablet_manager.cc:1367] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20240501 08:12:31.911228 18705 raft_consensus.cc:348] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:31.911567 18705 raft_consensus.cc:374] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.911725 18705 raft_consensus.cc:729] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.912036 18705 consensus_queue.cc:260] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:31.912775 18705 ts_tablet_manager.cc:1398] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:31.913446 18705 tablet_bootstrap.cc:492] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:31.921869 18705 tablet_bootstrap.cc:654] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.934509 18702 tablet_bootstrap.cc:492] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:31.934826 18702 ts_tablet_manager.cc:1367] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.002s
I20240501 08:12:31.937757 18702 raft_consensus.cc:348] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:31.938221 18702 raft_consensus.cc:374] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.938351 18702 raft_consensus.cc:729] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.939489 18705 tablet_bootstrap.cc:492] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:31.939936 18702 consensus_queue.cc:260] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:31.941990 18702 ts_tablet_manager.cc:1398] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 08:12:31.942332 18705 ts_tablet_manager.cc:1367] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.005s
I20240501 08:12:31.943712 18705 raft_consensus.cc:348] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.944185 18705 raft_consensus.cc:374] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.944348 18705 raft_consensus.cc:729] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.944773 18705 consensus_queue.cc:260] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:31.945875 18705 ts_tablet_manager.cc:1398] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:31.950867 18705 tablet_bootstrap.cc:492] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:31.953387 18705 tablet_bootstrap.cc:654] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:31.958750 12831 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:31.958942 12831 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:31.962817 12831 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 76fdcb09c5d242058b9954832855f5cd
I20240501 08:12:31.963660 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fdcb09c5d242058b9954832855f5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:31 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:31.963939 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fdcb09c5d242058b9954832855f5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:31 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:31.964493 15735 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fdcb09c5d242058b9954832855f5cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:31 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:31.978869 18722 tablet_replica.cc:330] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:31.979571 18722 raft_consensus.cc:2229] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:31.980096 18722 raft_consensus.cc:2258] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:31.981653 18705 tablet_bootstrap.cc:492] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:31.981992 18705 ts_tablet_manager.cc:1367] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.005s
I20240501 08:12:31.982923 18705 raft_consensus.cc:348] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.984923 18705 raft_consensus.cc:374] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:31.985109 18705 raft_consensus.cc:729] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:31.985394 18705 consensus_queue.cc:260] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:31.986158 18705 ts_tablet_manager.cc:1398] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:12:31.986399 18721 tablet_replica.cc:330] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:31.987416 18722 ts_tablet_manager.cc:1875] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:31.992075 18721 raft_consensus.cc:2229] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:31.992451 18721 raft_consensus.cc:2258] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:31.995386 18721 ts_tablet_manager.cc:1875] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:31.997916 18217 raft_consensus.cc:482] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:31.998095 18217 raft_consensus.cc:504] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:31.999662 18217 leader_election.cc:288] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:32.000161 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321bf6596ff24ec7af79e0510eadedce" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:32.000439 16025 raft_consensus.cc:2454] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:32.001154 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321bf6596ff24ec7af79e0510eadedce" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:32.001340 12974 raft_consensus.cc:2454] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:32.001835 15691 leader_election.cc:302] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:32.002929 18307 raft_consensus.cc:2781] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:32.003063 18307 raft_consensus.cc:482] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:32.003157 18307 raft_consensus.cc:3037] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.004274 18723 tablet_replica.cc:330] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:32.006210 18307 raft_consensus.cc:504] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:32.009140 18307 leader_election.cc:288] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:32.009555 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321bf6596ff24ec7af79e0510eadedce" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:32.009754 12974 raft_consensus.cc:3037] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.010090 18723 raft_consensus.cc:2229] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:32.010425 18723 raft_consensus.cc:2258] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.011462 18723 ts_tablet_manager.cc:1875] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:32.012372 18436 raft_consensus.cc:482] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:32.012578 18436 raft_consensus.cc:504] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:32.013401 18436 leader_election.cc:288] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:32.016481 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0020baddc4f47b625edc2e1180e51" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:32.016722 16024 raft_consensus.cc:2454] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:32.017925 15825 leader_election.cc:302] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:32.018507 12974 raft_consensus.cc:2454] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:32.019351 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321bf6596ff24ec7af79e0510eadedce" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:32.019587 16025 raft_consensus.cc:3037] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.020457 18436 raft_consensus.cc:2781] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:32.020556 18436 raft_consensus.cc:482] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:32.020648 18436 raft_consensus.cc:3037] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.023259 18436 raft_consensus.cc:504] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:32.024281 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0020baddc4f47b625edc2e1180e51" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:32.024506 16024 raft_consensus.cc:3037] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.024607 15692 leader_election.cc:302] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:32.024726 18339 raft_consensus.cc:482] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:32.025025 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0020baddc4f47b625edc2e1180e51" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:32.024953 18339 raft_consensus.cc:504] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:32.025753 18436 leader_election.cc:288] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:32.026072 18339 leader_election.cc:288] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:32.026356 18307 raft_consensus.cc:2781] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:32.026523 18307 raft_consensus.cc:686] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:32.026896 18307 consensus_queue.cc:237] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:32.027843 16024 raft_consensus.cc:2454] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:32.028427 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0020baddc4f47b625edc2e1180e51" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:32.028915 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0020baddc4f47b625edc2e1180e51" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:32.029299 16023 raft_consensus.cc:2379] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cc32f6e2d49477f9a23ce6e444039fb in current term 1: Already voted for candidate 4d516916eeac468db514eeb928f06190 in this term.
I20240501 08:12:32.029484 15825 leader_election.cc:302] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:32.030550 16025 raft_consensus.cc:2454] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:32.030680 16154 raft_consensus.cc:2454] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:32.031670 18436 raft_consensus.cc:2781] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:32.031837 18436 raft_consensus.cc:686] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:32.032107 18436 consensus_queue.cc:237] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:32.050804 18722 ts_tablet_manager.cc:1888] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:32.051648 12829 catalog_manager.cc:5484] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:32.055084 18722 log.cc:1196] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/76fdcb09c5d242058b9954832855f5cd
I20240501 08:12:32.057560 18722 ts_tablet_manager.cc:1909] T 76fdcb09c5d242058b9954832855f5cd P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:32.064149 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 76fdcb09c5d242058b9954832855f5cd (table rebalancer_dynamic_table_new_name [id=53ea9064c91d47dbb10c13383f524daa]) successfully deleted
I20240501 08:12:32.064389 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b0020baddc4f47b625edc2e1180e51" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:32.065711 16090 leader_election.cc:302] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161
I20240501 08:12:32.066265 18339 raft_consensus.cc:3037] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.066154 12829 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:32.069212 18339 raft_consensus.cc:2726] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:32.080106 16138 tablet_service.cc:1467] Processing CreateTablet for tablet a2d7a34af709440fb062746fdb3d12d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=154abd2c3a924147818e417e91e89159]), partition=
I20240501 08:12:32.080569 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d7a34af709440fb062746fdb3d12d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.081035 13084 tablet_service.cc:1467] Processing CreateTablet for tablet a2d7a34af709440fb062746fdb3d12d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=154abd2c3a924147818e417e91e89159]), partition=
I20240501 08:12:32.081367 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d7a34af709440fb062746fdb3d12d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.086768 18706 tablet_bootstrap.cc:492] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:32.088735 18706 tablet_bootstrap.cc:654] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.094974 18721 ts_tablet_manager.cc:1888] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:32.097664 12831 catalog_manager.cc:5484] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:32.101872 15872 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:32.095795 18721 log.cc:1196] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/76fdcb09c5d242058b9954832855f5cd
W20240501 08:12:32.112489 15826 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 55286.2us
I20240501 08:12:32.113847 18721 ts_tablet_manager.cc:1909] T 76fdcb09c5d242058b9954832855f5cd P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:32.114171 15872 tablet_service.cc:1467] Processing CreateTablet for tablet a2d7a34af709440fb062746fdb3d12d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=154abd2c3a924147818e417e91e89159]), partition=
I20240501 08:12:32.114595 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2d7a34af709440fb062746fdb3d12d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.123709 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 76fdcb09c5d242058b9954832855f5cd (table rebalancer_dynamic_table_new_name [id=53ea9064c91d47dbb10c13383f524daa]) successfully deleted
I20240501 08:12:32.124668 18704 tablet_bootstrap.cc:492] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:32.126408 18723 ts_tablet_manager.cc:1888] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:32.126538 18723 log.cc:1196] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/76fdcb09c5d242058b9954832855f5cd
I20240501 08:12:32.126757 18704 tablet_bootstrap.cc:654] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.126991 18723 ts_tablet_manager.cc:1909] T 76fdcb09c5d242058b9954832855f5cd P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:32.129513 18704 tablet_bootstrap.cc:492] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:32.129756 18704 ts_tablet_manager.cc:1367] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 08:12:32.131337 18736 tablet_bootstrap.cc:492] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:32.131292 18704 raft_consensus.cc:348] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:32.131650 18704 raft_consensus.cc:374] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.131835 18704 raft_consensus.cc:729] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.132282 18704 consensus_queue.cc:260] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:32.134723 18704 ts_tablet_manager.cc:1398] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:32.149276 18706 tablet_bootstrap.cc:492] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:32.149477 18706 ts_tablet_manager.cc:1367] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.002s
I20240501 08:12:32.150770 18706 raft_consensus.cc:348] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:32.151031 18706 raft_consensus.cc:374] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.151113 18706 raft_consensus.cc:729] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.152841 18706 consensus_queue.cc:260] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:32.155531 18706 ts_tablet_manager.cc:1398] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:12:32.160534 13344 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:32.164633 18736 tablet_bootstrap.cc:654] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.174461 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 76fdcb09c5d242058b9954832855f5cd (table rebalancer_dynamic_table_new_name [id=53ea9064c91d47dbb10c13383f524daa]) successfully deleted
I20240501 08:12:32.211553 18345 raft_consensus.cc:482] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:32.211784 18345 raft_consensus.cc:504] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:32.214357 18345 leader_election.cc:288] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:32.215058 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d241e087e754485cb80080754d27aef7" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:32.215297 12974 raft_consensus.cc:2454] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:32.216447 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d241e087e754485cb80080754d27aef7" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:32.216699 15892 raft_consensus.cc:2454] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:32.216820 15960 leader_election.cc:302] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:32.217682 18345 raft_consensus.cc:2781] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:32.217800 18345 raft_consensus.cc:482] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:32.217882 18345 raft_consensus.cc:3037] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.220870 18345 raft_consensus.cc:504] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:32.225716 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d241e087e754485cb80080754d27aef7" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:32.225926 12974 raft_consensus.cc:3037] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.226886 18736 tablet_bootstrap.cc:492] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:32.227018 18736 ts_tablet_manager.cc:1367] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.096s user 0.000s sys 0.006s
I20240501 08:12:32.227757 18736 raft_consensus.cc:348] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:32.227981 18736 raft_consensus.cc:374] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.228042 18736 raft_consensus.cc:729] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.228317 18736 consensus_queue.cc:260] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:32.228885 18736 ts_tablet_manager.cc:1398] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:32.229681 12974 raft_consensus.cc:2454] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:32.230202 15960 leader_election.cc:302] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:32.230468 18202 raft_consensus.cc:2781] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:32.230595 18202 raft_consensus.cc:686] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:32.230950 18202 consensus_queue.cc:237] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:32.238663 18345 leader_election.cc:288] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:32.248059 18339 raft_consensus.cc:482] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:32.248281 18339 raft_consensus.cc:504] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:32.249217 18339 leader_election.cc:288] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:32.250560 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d7a34af709440fb062746fdb3d12d1" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:32.250602 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d7a34af709440fb062746fdb3d12d1" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:32.250820 15892 raft_consensus.cc:2454] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:32.250881 13104 raft_consensus.cc:2454] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:32.251488 16090 leader_election.cc:302] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:32.252017 18339 raft_consensus.cc:2781] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:32.252141 18339 raft_consensus.cc:482] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:32.252259 18339 raft_consensus.cc:3037] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.255805 18339 raft_consensus.cc:504] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:32.257361 16138 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:32.261221 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d7a34af709440fb062746fdb3d12d1" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:32.261445 15892 raft_consensus.cc:3037] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.269369 18339 leader_election.cc:288] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:32.270259 15892 raft_consensus.cc:2454] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:32.270987 16090 leader_election.cc:302] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:32.272558 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2d7a34af709440fb062746fdb3d12d1" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:32.272797 13104 raft_consensus.cc:3037] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.272956 18339 raft_consensus.cc:2781] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:32.273535 18339 raft_consensus.cc:686] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:32.273847 18339 consensus_queue.cc:237] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:32.276091 13104 raft_consensus.cc:2454] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:32.281980 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d241e087e754485cb80080754d27aef7" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:32.282361 15892 raft_consensus.cc:3037] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:32.302208 15892 raft_consensus.cc:2454] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:32.321286 12829 catalog_manager.cc:5484] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:32.325542 12831 catalog_manager.cc:5484] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:32.333626 16004 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:32.361452 15735 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:32.378450 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:32.385039 13213 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:32.393760 12954 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:32.428944 12831 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:32.446210 15892 raft_consensus.cc:1261] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:32.447335 18339 consensus_queue.cc:1035] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.447583 13104 raft_consensus.cc:1261] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:32.453318 18430 consensus_queue.cc:1035] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.469226 12831 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet a2d7a34af709440fb062746fdb3d12d1 (table rebalancer_dynamic_table [id=154abd2c3a924147818e417e91e89159]). Expected version 1 got 0
I20240501 08:12:32.473594 18761 tablet.cc:1722] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:32.474643 18761 tablet.cc:1620] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:32.484892 18762 tablet.cc:1722] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:32.487910 12831 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet a2d7a34af709440fb062746fdb3d12d1 (table rebalancer_dynamic_table [id=154abd2c3a924147818e417e91e89159]). Expected version 1 got 0
I20240501 08:12:32.488333 18762 tablet.cc:1620] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:32.494673 12831 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=154abd2c3a924147818e417e91e89159] alter complete (version 1)
I20240501 08:12:32.502818 18307 consensus_queue.cc:1035] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.513406 12833 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet a2d7a34af709440fb062746fdb3d12d1 (table rebalancer_dynamic_table [id=154abd2c3a924147818e417e91e89159]). Expected version 1 got 0
I20240501 08:12:32.521855 18768 tablet.cc:1722] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:32.523285 18768 tablet.cc:1620] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:32.526228 12833 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet a2d7a34af709440fb062746fdb3d12d1 (table rebalancer_dynamic_table [id=154abd2c3a924147818e417e91e89159]). Expected version 1 got 0
I20240501 08:12:32.536131 18217 consensus_queue.cc:1035] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.577256 18439 consensus_queue.cc:1035] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.585422 18439 consensus_queue.cc:1035] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.603457 12833 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:32.618851 18761 tablet.cc:1722] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:32.619395 18761 tablet.cc:1620] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:32.623083 18762 tablet.cc:1722] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:32.623560 18762 tablet.cc:1620] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:32.625958 18768 tablet.cc:1722] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:32.626467 18768 tablet.cc:1620] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:32.640097 12831 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=154abd2c3a924147818e417e91e89159] alter complete (version 2)
I20240501 08:12:32.648967 13104 consensus_queue.cc:237] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:32.651329 13235 raft_consensus.cc:1261] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:32.651917 16025 raft_consensus.cc:1261] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:32.652505 18458 consensus_queue.cc:1035] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:32.652734 18697 consensus_queue.cc:1035] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:32.656455 18697 raft_consensus.cc:2932] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:32.659267 13235 raft_consensus.cc:2932] T 911cc47749484e88882c0efa75fc6396 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:32.660894 16025 raft_consensus.cc:2932] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:32.664680 12831 catalog_manager.cc:5484] T 911cc47749484e88882c0efa75fc6396 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:32.674921 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet 911cc47749484e88882c0efa75fc6396 with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:32.675778 18794 tablet_replica.cc:330] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:32.676026 18794 raft_consensus.cc:2229] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:32.677155 18794 raft_consensus.cc:2258] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.677830 18794 ts_tablet_manager.cc:1875] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:32.683717 18794 ts_tablet_manager.cc:1888] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:32.683843 18794 log.cc:1196] T 911cc47749484e88882c0efa75fc6396 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/911cc47749484e88882c0efa75fc6396
I20240501 08:12:32.684553 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 911cc47749484e88882c0efa75fc6396 (table rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]) successfully deleted
I20240501 08:12:32.712625 18202 consensus_queue.cc:1035] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.716861 18202 consensus_queue.cc:1035] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:32.747038 12829 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:32.756502 18761 tablet.cc:1722] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:32.760241 18768 tablet.cc:1722] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:32.770263 12831 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=154abd2c3a924147818e417e91e89159] alter complete (version 3)
I20240501 08:12:32.773892 18762 tablet.cc:1722] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:32.872875 12827 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:32.873118 12827 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:32.881762 12827 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet a2d7a34af709440fb062746fdb3d12d1
I20240501 08:12:32.882541 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet a2d7a34af709440fb062746fdb3d12d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:32 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:32.883467 18839 tablet_replica.cc:330] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:32.883781 18839 raft_consensus.cc:2229] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:32.884240 18839 raft_consensus.cc:2258] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.885329 18839 ts_tablet_manager.cc:1875] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:32.885612 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet a2d7a34af709440fb062746fdb3d12d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:32 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:32.887594 12824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:32.888115 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet a2d7a34af709440fb062746fdb3d12d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:32 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:32.888741 18841 tablet_replica.cc:330] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:32.888952 18841 raft_consensus.cc:2229] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:32.889212 18841 raft_consensus.cc:2258] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.889990 18841 ts_tablet_manager.cc:1875] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:32.898324 18841 ts_tablet_manager.cc:1888] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:32.898551 18841 log.cc:1196] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/a2d7a34af709440fb062746fdb3d12d1
I20240501 08:12:32.898674 18839 ts_tablet_manager.cc:1888] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:32.899101 18841 ts_tablet_manager.cc:1909] T a2d7a34af709440fb062746fdb3d12d1 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:32.899176 18839 log.cc:1196] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/a2d7a34af709440fb062746fdb3d12d1
I20240501 08:12:32.899701 18839 ts_tablet_manager.cc:1909] T a2d7a34af709440fb062746fdb3d12d1 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:32.900280 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet a2d7a34af709440fb062746fdb3d12d1 (table rebalancer_dynamic_table_new_name [id=154abd2c3a924147818e417e91e89159]) successfully deleted
I20240501 08:12:32.902354 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet a2d7a34af709440fb062746fdb3d12d1 (table rebalancer_dynamic_table_new_name [id=154abd2c3a924147818e417e91e89159]) successfully deleted
I20240501 08:12:32.903084 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 3549f34428b34ea3920f0f0f20c24524 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.903174 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 3549f34428b34ea3920f0f0f20c24524 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.903584 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3549f34428b34ea3920f0f0f20c24524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.903609 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3549f34428b34ea3920f0f0f20c24524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.903719 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6040f5ac944fdd9a4004f43f20d40d (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.904094 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6040f5ac944fdd9a4004f43f20d40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.907833 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 1eac68af4bc3473487bf91670cb48223 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.908262 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eac68af4bc3473487bf91670cb48223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.908372 18842 tablet_bootstrap.cc:492] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:32.908730 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 1eac68af4bc3473487bf91670cb48223 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.909173 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eac68af4bc3473487bf91670cb48223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.911149 18842 tablet_bootstrap.cc:654] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.911535 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6040f5ac944fdd9a4004f43f20d40d (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.911959 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6040f5ac944fdd9a4004f43f20d40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.915551 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 0c6040f5ac944fdd9a4004f43f20d40d (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.915594 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 1eac68af4bc3473487bf91670cb48223 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.915999 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c6040f5ac944fdd9a4004f43f20d40d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.919049 15735 tablet_service.cc:1467] Processing CreateTablet for tablet 3549f34428b34ea3920f0f0f20c24524 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:32.921474 15735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3549f34428b34ea3920f0f0f20c24524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.921924 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eac68af4bc3473487bf91670cb48223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:32.931152 18846 tablet_bootstrap.cc:492] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:32.937321 18846 tablet_bootstrap.cc:654] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.937382 18840 tablet_replica.cc:330] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:32.937590 18840 raft_consensus.cc:2229] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:32.938107 18840 raft_consensus.cc:2258] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:32.938838 18847 tablet_bootstrap.cc:492] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:32.943585 18840 ts_tablet_manager.cc:1875] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:32.944437 18848 tablet_bootstrap.cc:492] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:32.946630 18845 tablet_bootstrap.cc:492] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:32.947160 18844 tablet_bootstrap.cc:492] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:32.950423 18844 tablet_bootstrap.cc:654] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.952272 18848 tablet_bootstrap.cc:654] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.954465 18847 tablet_bootstrap.cc:654] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.961180 18845 tablet_bootstrap.cc:654] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:32.963264 18844 tablet_bootstrap.cc:492] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:32.963372 18847 tablet_bootstrap.cc:492] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:32.964532 18847 ts_tablet_manager.cc:1367] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240501 08:12:32.965221 18844 ts_tablet_manager.cc:1367] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 08:12:32.965350 18840 ts_tablet_manager.cc:1888] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:32.965500 18847 raft_consensus.cc:348] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:32.966079 18844 raft_consensus.cc:348] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:32.967638 18844 raft_consensus.cc:374] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.967777 18844 raft_consensus.cc:729] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.968060 18844 consensus_queue.cc:260] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:32.968832 18844 ts_tablet_manager.cc:1398] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:32.971822 18847 raft_consensus.cc:374] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.971982 18847 raft_consensus.cc:729] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.973341 18847 consensus_queue.cc:260] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:32.974478 18848 tablet_bootstrap.cc:492] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:32.974692 18848 ts_tablet_manager.cc:1367] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.000s
I20240501 08:12:32.976034 18848 raft_consensus.cc:348] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:32.976444 18848 raft_consensus.cc:374] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:32.976588 18848 raft_consensus.cc:729] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:32.976989 18848 consensus_queue.cc:260] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:32.977998 18848 ts_tablet_manager.cc:1398] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:32.982144 18847 ts_tablet_manager.cc:1398] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240501 08:12:32.989723 18842 tablet_bootstrap.cc:492] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:32.990145 18842 ts_tablet_manager.cc:1367] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.082s user 0.013s sys 0.000s
I20240501 08:12:32.991371 18842 raft_consensus.cc:348] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:32.994117 12831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:33.002180 18846 tablet_bootstrap.cc:492] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:33.002436 18846 ts_tablet_manager.cc:1367] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.002s
I20240501 08:12:33.006919 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 40175b60e303488d8f7851a9a59fe900 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cff2596f87d04cd88b456b165c97ed3d]), partition=
I20240501 08:12:33.007066 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 40175b60e303488d8f7851a9a59fe900 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cff2596f87d04cd88b456b165c97ed3d]), partition=
I20240501 08:12:33.007349 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40175b60e303488d8f7851a9a59fe900. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.007392 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40175b60e303488d8f7851a9a59fe900. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.014657 18844 tablet_bootstrap.cc:492] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:33.015974 18844 tablet_bootstrap.cc:654] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.018002 18842 raft_consensus.cc:374] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.018249 18842 raft_consensus.cc:729] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.018541 18842 consensus_queue.cc:260] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:33.020740 18842 ts_tablet_manager.cc:1398] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240501 08:12:33.021124 18842 tablet_bootstrap.cc:492] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:33.022820 18842 tablet_bootstrap.cc:654] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.034473 18844 tablet_bootstrap.cc:492] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:33.034719 18844 ts_tablet_manager.cc:1367] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20240501 08:12:33.026065 18339 raft_consensus.cc:482] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.035266 18339 raft_consensus.cc:504] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:33.035455 18844 raft_consensus.cc:348] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:33.035652 18844 raft_consensus.cc:374] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.035705 18844 raft_consensus.cc:729] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.036011 18339 leader_election.cc:288] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:33.035936 18844 consensus_queue.cc:260] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:33.036916 18844 ts_tablet_manager.cc:1398] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:33.042634 16138 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:33.048048 13344 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:33.051870 18848 tablet_bootstrap.cc:492] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:33.053284 18848 tablet_bootstrap.cc:654] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.053892 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3549f34428b34ea3920f0f0f20c24524" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
W20240501 08:12:33.055132 16092 leader_election.cc:341] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:33.055357 18846 raft_consensus.cc:348] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:33.055713 18846 raft_consensus.cc:374] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.055878 18846 raft_consensus.cc:729] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.064752 18751 raft_consensus.cc:482] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.064980 18751 raft_consensus.cc:504] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:33.065862 18751 leader_election.cc:288] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:33.067298 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6040f5ac944fdd9a4004f43f20d40d" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:33.067353 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6040f5ac944fdd9a4004f43f20d40d" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:33.067684 13104 raft_consensus.cc:2454] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:33.068775 13170 leader_election.cc:302] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:33.074257 18751 raft_consensus.cc:2781] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:33.074393 18751 raft_consensus.cc:482] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:33.074466 18751 raft_consensus.cc:3037] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.081391 18751 raft_consensus.cc:504] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:33.083050 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6040f5ac944fdd9a4004f43f20d40d" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:33.086668 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c6040f5ac944fdd9a4004f43f20d40d" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:33.086961 13104 raft_consensus.cc:3037] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.090888 13104 raft_consensus.cc:2454] T 0c6040f5ac944fdd9a4004f43f20d40d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:33.096246 13170 leader_election.cc:302] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters: 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:33.096721 18619 raft_consensus.cc:2781] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:33.096876 18619 raft_consensus.cc:686] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:33.097226 18619 consensus_queue.cc:237] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:33.119794 18751 leader_election.cc:288] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:33.130939 18846 consensus_queue.cc:260] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:33.131628 15756 raft_consensus.cc:2454] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:33.131925 18846 ts_tablet_manager.cc:1398] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.129s user 0.002s sys 0.000s
I20240501 08:12:33.133481 18846 tablet_bootstrap.cc:492] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:33.135378 18846 tablet_bootstrap.cc:654] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.136931 18840 log.cc:1196] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/a2d7a34af709440fb062746fdb3d12d1
I20240501 08:12:33.138338 18840 ts_tablet_manager.cc:1909] T a2d7a34af709440fb062746fdb3d12d1 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:33.005491 15870 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:33.006477 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 40175b60e303488d8f7851a9a59fe900 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cff2596f87d04cd88b456b165c97ed3d]), partition=
I20240501 08:12:33.038671 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3549f34428b34ea3920f0f0f20c24524" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:32.971091 18845 tablet_bootstrap.cc:492] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:33.139628 18845 ts_tablet_manager.cc:1367] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.193s user 0.004s sys 0.000s
I20240501 08:12:33.140396 18845 raft_consensus.cc:348] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:33.140566 18845 raft_consensus.cc:374] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.140671 18845 raft_consensus.cc:729] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.140939 18845 consensus_queue.cc:260] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:33.141639 18845 ts_tablet_manager.cc:1398] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240501 08:12:33.142019 16090 leader_election.cc:341] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:33.142166 16090 leader_election.cc:302] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:33.142617 18339 raft_consensus.cc:2726] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:33.139467 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40175b60e303488d8f7851a9a59fe900. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.174456 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:33.179385 18842 tablet_bootstrap.cc:492] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:33.179594 18842 ts_tablet_manager.cc:1367] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.159s user 0.005s sys 0.014s
I20240501 08:12:33.180321 18845 tablet_bootstrap.cc:492] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:33.182394 18845 tablet_bootstrap.cc:654] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.183795 18842 raft_consensus.cc:348] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:33.184044 18842 raft_consensus.cc:374] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.184118 18842 raft_consensus.cc:729] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.184391 18842 consensus_queue.cc:260] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:33.185510 18842 ts_tablet_manager.cc:1398] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:33.185894 18815 raft_consensus.cc:482] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.186018 18815 raft_consensus.cc:504] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:33.186878 18815 leader_election.cc:288] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:33.187403 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eac68af4bc3473487bf91670cb48223" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:33.187703 13363 raft_consensus.cc:2454] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:33.189033 16092 leader_election.cc:302] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:33.189603 18339 raft_consensus.cc:2781] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:33.189695 18339 raft_consensus.cc:482] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:33.189797 18339 raft_consensus.cc:3037] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.193594 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eac68af4bc3473487bf91670cb48223" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
W20240501 08:12:33.194651 16092 leader_election.cc:341] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:33.200240 18339 raft_consensus.cc:504] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:33.202816 18339 leader_election.cc:288] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:33.203689 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eac68af4bc3473487bf91670cb48223" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:33.203922 13363 raft_consensus.cc:3037] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.207319 13363 raft_consensus.cc:2454] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:33.209036 16092 leader_election.cc:302] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:33.209960 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1eac68af4bc3473487bf91670cb48223" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
W20240501 08:12:33.210911 16092 leader_election.cc:341] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:33.211186 12954 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:33.213357 18339 raft_consensus.cc:2781] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:33.213696 18339 raft_consensus.cc:686] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:33.214058 18339 consensus_queue.cc:237] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:33.219967 18848 tablet_bootstrap.cc:492] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:33.220221 18848 ts_tablet_manager.cc:1367] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.168s user 0.035s sys 0.061s
I20240501 08:12:33.221256 18848 raft_consensus.cc:348] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:33.221784 18848 raft_consensus.cc:374] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.221932 18848 raft_consensus.cc:729] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.222302 18848 consensus_queue.cc:260] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:33.223023 18848 ts_tablet_manager.cc:1398] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:33.230458 18846 tablet_bootstrap.cc:492] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:33.230770 18846 ts_tablet_manager.cc:1367] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.097s user 0.001s sys 0.035s
I20240501 08:12:33.231842 18846 raft_consensus.cc:348] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:33.232179 18846 raft_consensus.cc:374] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.232342 18846 raft_consensus.cc:729] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.232728 18846 consensus_queue.cc:260] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:33.234632 18846 ts_tablet_manager.cc:1398] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:33.234961 18846 tablet_bootstrap.cc:492] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:33.236707 18846 tablet_bootstrap.cc:654] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.245561 18307 raft_consensus.cc:482] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.245769 18307 raft_consensus.cc:504] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:33.246753 18307 leader_election.cc:288] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:33.247850 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3549f34428b34ea3920f0f0f20c24524" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:33.248173 15893 raft_consensus.cc:2454] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:33.249830 15689 leader_election.cc:302] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:33.250130 18307 raft_consensus.cc:2781] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:33.250252 18307 raft_consensus.cc:482] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:33.250355 18307 raft_consensus.cc:3037] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.251345 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3549f34428b34ea3920f0f0f20c24524" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:33.251560 16153 raft_consensus.cc:2454] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:33.256040 18307 raft_consensus.cc:504] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:33.257117 18307 leader_election.cc:288] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:33.260598 13213 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:33.261215 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3549f34428b34ea3920f0f0f20c24524" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:33.261457 15893 raft_consensus.cc:3037] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.265374 15893 raft_consensus.cc:2454] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:33.266021 15689 leader_election.cc:302] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:33.266480 18217 raft_consensus.cc:2781] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:33.266521 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3549f34428b34ea3920f0f0f20c24524" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:33.266634 18217 raft_consensus.cc:686] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:33.266719 16153 raft_consensus.cc:3037] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.266875 18217 consensus_queue.cc:237] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:33.271023 16153 raft_consensus.cc:2454] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:33.279175 12831 catalog_manager.cc:5484] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:33.304091 18846 tablet_bootstrap.cc:492] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:33.308954 18846 ts_tablet_manager.cc:1367] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.011s
I20240501 08:12:33.311115 18846 raft_consensus.cc:348] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:33.312201 18846 raft_consensus.cc:374] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.312515 18846 raft_consensus.cc:729] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.312752 18846 consensus_queue.cc:260] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:33.313326 18846 ts_tablet_manager.cc:1398] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 08:12:33.361683 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet a2d7a34af709440fb062746fdb3d12d1 (table rebalancer_dynamic_table_new_name [id=154abd2c3a924147818e417e91e89159]) successfully deleted
W20240501 08:12:33.372009 15689 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 105969us
I20240501 08:12:33.377261 12831 catalog_manager.cc:5484] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:33.379971 12824 catalog_manager.cc:5484] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:33.383783 16004 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:33.388619 18845 tablet_bootstrap.cc:492] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:33.388891 18845 ts_tablet_manager.cc:1367] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.209s user 0.004s sys 0.006s
I20240501 08:12:33.389945 18845 raft_consensus.cc:348] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:33.390265 18845 raft_consensus.cc:374] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:33.390399 18845 raft_consensus.cc:729] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:33.390709 18845 consensus_queue.cc:260] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:33.391441 18845 ts_tablet_manager.cc:1398] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:33.392064 15733 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240501 08:12:33.392525 15691 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.66:35509, user_credentials={real_user=slave}} blocked reactor thread for 123497us
I20240501 08:12:33.450318 18883 raft_consensus.cc:482] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:33.450497 18883 raft_consensus.cc:504] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:33.451936 18883 leader_election.cc:288] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:33.452030 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40175b60e303488d8f7851a9a59fe900" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:33.452280 13104 raft_consensus.cc:2454] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:33.452381 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40175b60e303488d8f7851a9a59fe900" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:33.452597 15893 raft_consensus.cc:2454] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:33.452908 13300 leader_election.cc:302] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:33.453747 18883 raft_consensus.cc:2781] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:33.453886 18883 raft_consensus.cc:482] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:33.454020 18883 raft_consensus.cc:3037] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.456764 18883 raft_consensus.cc:504] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:33.457367 18883 leader_election.cc:288] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:33.457810 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40175b60e303488d8f7851a9a59fe900" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:33.458024 13104 raft_consensus.cc:3037] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.458372 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40175b60e303488d8f7851a9a59fe900" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:33.458552 15893 raft_consensus.cc:3037] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.461375 15893 raft_consensus.cc:2454] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:33.461478 13104 raft_consensus.cc:2454] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:33.461746 13298 leader_election.cc:302] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters:
I20240501 08:12:33.462085 18883 raft_consensus.cc:2781] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:33.462625 18883 raft_consensus.cc:686] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:33.462905 18883 consensus_queue.cc:237] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:33.466570 12824 catalog_manager.cc:5484] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:33.536785 15756 raft_consensus.cc:3037] T 0c6040f5ac944fdd9a4004f43f20d40d P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.540897 18619 consensus_queue.cc:1035] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.557125 18451 consensus_queue.cc:1035] T 0c6040f5ac944fdd9a4004f43f20d40d P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.598320 13363 consensus_queue.cc:237] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:33.601833 16153 raft_consensus.cc:1261] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:33.601833 13235 raft_consensus.cc:1261] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:33.602447 15756 raft_consensus.cc:3037] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:33.602444 18883 consensus_queue.cc:1035] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:33.602761 18310 consensus_queue.cc:1035] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:33.605213 18883 raft_consensus.cc:2932] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:33.606648 18815 consensus_queue.cc:1035] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.609431 13235 raft_consensus.cc:2932] T 9348009ae1444e73b6120161309dc4a2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:33.614573 12824 catalog_manager.cc:5484] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:33.620555 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 9348009ae1444e73b6120161309dc4a2 with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:33.627473 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:33.634039 18912 tablet_replica.cc:330] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:33.634289 18912 raft_consensus.cc:2229] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:33.634524 18912 raft_consensus.cc:2258] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:33.635200 18912 ts_tablet_manager.cc:1875] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:33.638500 15893 raft_consensus.cc:1261] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:33.639156 18339 consensus_queue.cc:1035] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.639456 18883 consensus_queue.cc:1035] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.641022 13104 raft_consensus.cc:1261] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:33.642320 18310 consensus_queue.cc:1035] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.643373 18912 ts_tablet_manager.cc:1888] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:33.643478 18912 log.cc:1196] T 9348009ae1444e73b6120161309dc4a2 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/9348009ae1444e73b6120161309dc4a2
I20240501 08:12:33.644389 16154 raft_consensus.cc:2932] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:33.648583 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 9348009ae1444e73b6120161309dc4a2 (table rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]) successfully deleted
I20240501 08:12:33.681144 18922 tablet.cc:1722] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:33.682763 12824 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet 40175b60e303488d8f7851a9a59fe900 (table rebalancer_dynamic_table [id=cff2596f87d04cd88b456b165c97ed3d]). Expected version 1 got 0
I20240501 08:12:33.683837 18922 tablet.cc:1620] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:33.694473 18924 tablet.cc:1722] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:33.696319 18924 tablet.cc:1620] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:33.700978 12826 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet 40175b60e303488d8f7851a9a59fe900 (table rebalancer_dynamic_table [id=cff2596f87d04cd88b456b165c97ed3d]). Expected version 1 got 0
I20240501 08:12:33.702391 12827 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 40175b60e303488d8f7851a9a59fe900 (table rebalancer_dynamic_table [id=cff2596f87d04cd88b456b165c97ed3d]). Expected version 1 got 0
I20240501 08:12:33.708596 12824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=cff2596f87d04cd88b456b165c97ed3d] alter complete (version 1)
I20240501 08:12:33.717687 18927 tablet.cc:1722] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:33.718286 18927 tablet.cc:1620] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:33.723385 12824 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 40175b60e303488d8f7851a9a59fe900 (table rebalancer_dynamic_table [id=cff2596f87d04cd88b456b165c97ed3d]). Expected version 1 got 0
I20240501 08:12:33.749296 18882 consensus_queue.cc:1035] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.755435 18882 consensus_queue.cc:1035] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:33.817772 12827 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:33.838747 18922 tablet.cc:1722] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:33.839298 18922 tablet.cc:1620] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:33.839783 18924 tablet.cc:1722] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:33.840266 18924 tablet.cc:1620] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:33.847115 18927 tablet.cc:1722] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:33.847644 18927 tablet.cc:1620] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:33.861858 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=cff2596f87d04cd88b456b165c97ed3d] alter complete (version 2)
I20240501 08:12:33.907244 12827 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:33.923184 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8ec0d302644d378c54e4871f85159a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.923421 16004 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8ec0d302644d378c54e4871f85159a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.923794 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8ec0d302644d378c54e4871f85159a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.923840 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8ec0d302644d378c54e4871f85159a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.924047 13344 tablet_service.cc:1467] Processing CreateTablet for tablet f305a6e24ab1436b895db037f5d72870 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.924423 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f305a6e24ab1436b895db037f5d72870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.927592 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 3c8ec0d302644d378c54e4871f85159a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.928174 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8ec0d302644d378c54e4871f85159a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.930178 15870 tablet_service.cc:1467] Processing CreateTablet for tablet f305a6e24ab1436b895db037f5d72870 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.930552 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f305a6e24ab1436b895db037f5d72870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.930544 16003 tablet_service.cc:1467] Processing CreateTablet for tablet fa10a6f53f7f48cf9410c80e1056439d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.930913 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa10a6f53f7f48cf9410c80e1056439d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.931131 12954 tablet_service.cc:1467] Processing CreateTablet for tablet fa10a6f53f7f48cf9410c80e1056439d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.931515 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa10a6f53f7f48cf9410c80e1056439d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.933208 16137 tablet_service.cc:1467] Processing CreateTablet for tablet f305a6e24ab1436b895db037f5d72870 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.933601 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f305a6e24ab1436b895db037f5d72870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.937546 15871 tablet_service.cc:1467] Processing CreateTablet for tablet fa10a6f53f7f48cf9410c80e1056439d (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:33.937935 15871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa10a6f53f7f48cf9410c80e1056439d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:33.960067 18971 tablet_bootstrap.cc:492] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:33.962517 18975 tablet_bootstrap.cc:492] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:33.968942 12824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
W20240501 08:12:33.974067 12909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 52387.2us
I20240501 08:12:33.977142 18974 tablet_bootstrap.cc:492] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:33.981685 18971 tablet_bootstrap.cc:654] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.983201 18975 tablet_bootstrap.cc:654] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:33.984027 18972 tablet_bootstrap.cc:492] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:33.998275 18922 tablet.cc:1722] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:34.000123 18927 tablet.cc:1722] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:34.008785 18924 tablet.cc:1722] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:34.012334 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=cff2596f87d04cd88b456b165c97ed3d] alter complete (version 3)
I20240501 08:12:34.027684 18975 tablet_bootstrap.cc:492] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:33.983657 18973 tablet_bootstrap.cc:492] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:34.038014 18974 tablet_bootstrap.cc:654] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.040006 18973 tablet_bootstrap.cc:654] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.049382 18975 ts_tablet_manager.cc:1367] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.000s
I20240501 08:12:34.052480 18972 tablet_bootstrap.cc:654] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.052748 18971 tablet_bootstrap.cc:492] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:34.055056 18971 ts_tablet_manager.cc:1367] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.000s
I20240501 08:12:34.050908 18975 raft_consensus.cc:348] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.058938 18975 raft_consensus.cc:374] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.059075 18975 raft_consensus.cc:729] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.059412 18975 consensus_queue.cc:260] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.060027 18975 ts_tablet_manager.cc:1398] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240501 08:12:34.058454 18971 raft_consensus.cc:348] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.060813 18971 raft_consensus.cc:374] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.068784 18971 raft_consensus.cc:729] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.069205 18971 consensus_queue.cc:260] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.070775 18971 ts_tablet_manager.cc:1398] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240501 08:12:34.071457 13344 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:34.073812 18971 tablet_bootstrap.cc:492] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:34.076923 18971 tablet_bootstrap.cc:654] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.080451 15870 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:34.102808 18971 tablet_bootstrap.cc:492] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:34.103086 18971 ts_tablet_manager.cc:1367] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.014s
I20240501 08:12:34.105098 18971 raft_consensus.cc:348] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.105410 18971 raft_consensus.cc:374] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.105523 18971 raft_consensus.cc:729] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.105778 18971 consensus_queue.cc:260] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.106626 18971 ts_tablet_manager.cc:1398] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 08:12:34.107064 18971 tablet_bootstrap.cc:492] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:34.110455 18971 tablet_bootstrap.cc:654] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.117770 12832 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:34.118005 12832 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:34.123596 12832 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 40175b60e303488d8f7851a9a59fe900
I20240501 08:12:34.125605 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 40175b60e303488d8f7851a9a59fe900 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:34 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:34.128441 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet 40175b60e303488d8f7851a9a59fe900 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:34 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:34.128566 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet 40175b60e303488d8f7851a9a59fe900 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:34 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:34.129671 18990 tablet_replica.cc:330] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:34.135300 18990 raft_consensus.cc:2229] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:34.136281 18990 raft_consensus.cc:2258] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:34.138347 18990 ts_tablet_manager.cc:1875] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:34.147248 18991 tablet_replica.cc:330] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:34.147483 18992 tablet_replica.cc:330] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:34.147497 18991 raft_consensus.cc:2229] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:34.147774 18991 raft_consensus.cc:2258] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:34.148365 18992 raft_consensus.cc:2229] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:34.148896 18992 raft_consensus.cc:2258] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:34.154675 18991 ts_tablet_manager.cc:1875] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:34.160544 18974 tablet_bootstrap.cc:492] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:34.160857 18974 ts_tablet_manager.cc:1367] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.184s user 0.004s sys 0.000s
I20240501 08:12:34.161877 18974 raft_consensus.cc:348] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.162366 18974 raft_consensus.cc:374] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.162489 18974 raft_consensus.cc:729] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.162868 18974 consensus_queue.cc:260] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.163988 18974 ts_tablet_manager.cc:1398] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:34.168819 18990 ts_tablet_manager.cc:1888] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:34.170248 18973 tablet_bootstrap.cc:492] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:34.170554 18973 ts_tablet_manager.cc:1367] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.187s user 0.004s sys 0.000s
I20240501 08:12:34.171613 18973 raft_consensus.cc:348] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.171998 18973 raft_consensus.cc:374] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.172174 18973 raft_consensus.cc:729] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.172603 18973 consensus_queue.cc:260] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.173211 18990 log.cc:1196] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/40175b60e303488d8f7851a9a59fe900
I20240501 08:12:34.174962 18990 ts_tablet_manager.cc:1909] T 40175b60e303488d8f7851a9a59fe900 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:34.176205 18992 ts_tablet_manager.cc:1875] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:34.176971 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 40175b60e303488d8f7851a9a59fe900 (table rebalancer_dynamic_table_new_name [id=cff2596f87d04cd88b456b165c97ed3d]) successfully deleted
I20240501 08:12:34.190326 18972 tablet_bootstrap.cc:492] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:34.190591 18972 ts_tablet_manager.cc:1367] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.207s user 0.001s sys 0.004s
I20240501 08:12:34.191421 18972 raft_consensus.cc:348] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.191763 18972 raft_consensus.cc:374] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.191890 18972 raft_consensus.cc:729] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.192181 18972 consensus_queue.cc:260] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.195207 18972 ts_tablet_manager.cc:1398] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 08:12:34.218024 18991 ts_tablet_manager.cc:1888] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:34.219101 18973 ts_tablet_manager.cc:1398] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.048s user 0.002s sys 0.000s
I20240501 08:12:34.219449 18973 tablet_bootstrap.cc:492] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:34.221264 18973 tablet_bootstrap.cc:654] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.224705 18815 raft_consensus.cc:482] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:34.224905 18815 raft_consensus.cc:504] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.225776 18815 leader_election.cc:288] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:34.228502 12827 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:34.248129 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 74b3abba8f5d47a5ac65c5cce94459d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f6261e1204844c2a213b9c132786654]), partition=
I20240501 08:12:34.248582 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b3abba8f5d47a5ac65c5cce94459d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:34.255616 18974 tablet_bootstrap.cc:492] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:34.214267 16003 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:34.228129 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8ec0d302644d378c54e4871f85159a" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:34.256152 18972 tablet_bootstrap.cc:492] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
W20240501 08:12:34.256423 16092 leader_election.cc:341] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:34.257594 18974 tablet_bootstrap.cc:654] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.257990 18972 tablet_bootstrap.cc:654] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.260601 18973 tablet_bootstrap.cc:492] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:34.261322 18973 ts_tablet_manager.cc:1367] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.003s
I20240501 08:12:34.266754 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:34.227392 18971 tablet_bootstrap.cc:492] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:34.229252 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8ec0d302644d378c54e4871f85159a" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:34.269176 18992 ts_tablet_manager.cc:1888] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
W20240501 08:12:34.269177 16090 leader_election.cc:341] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:34.269337 16090 leader_election.cc:302] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161
I20240501 08:12:34.269752 18916 raft_consensus.cc:2726] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:34.270184 18971 ts_tablet_manager.cc:1367] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.163s user 0.022s sys 0.043s
I20240501 08:12:34.271086 18971 raft_consensus.cc:348] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.271314 18971 raft_consensus.cc:374] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.271421 18971 raft_consensus.cc:729] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.271677 18971 consensus_queue.cc:260] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.272207 18992 log.cc:1196] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/40175b60e303488d8f7851a9a59fe900
I20240501 08:12:34.272332 18971 ts_tablet_manager.cc:1398] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:34.273059 18992 ts_tablet_manager.cc:1909] T 40175b60e303488d8f7851a9a59fe900 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:34.274169 18991 log.cc:1196] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/40175b60e303488d8f7851a9a59fe900
I20240501 08:12:34.275223 18991 ts_tablet_manager.cc:1909] T 40175b60e303488d8f7851a9a59fe900 P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:34.277029 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 40175b60e303488d8f7851a9a59fe900 (table rebalancer_dynamic_table_new_name [id=cff2596f87d04cd88b456b165c97ed3d]) successfully deleted
I20240501 08:12:34.284756 18974 tablet_bootstrap.cc:492] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:34.284934 18974 ts_tablet_manager.cc:1367] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240501 08:12:34.285727 18974 raft_consensus.cc:348] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:34.285943 18974 raft_consensus.cc:374] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.285997 18974 raft_consensus.cc:729] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.286209 18974 consensus_queue.cc:260] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:34.288084 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 40175b60e303488d8f7851a9a59fe900 (table rebalancer_dynamic_table_new_name [id=cff2596f87d04cd88b456b165c97ed3d]) successfully deleted
I20240501 08:12:34.289180 18974 ts_tablet_manager.cc:1398] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:34.295614 18973 raft_consensus.cc:348] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.296021 18973 raft_consensus.cc:374] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.296142 18973 raft_consensus.cc:729] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.296360 18972 tablet_bootstrap.cc:492] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:34.296447 18973 consensus_queue.cc:260] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.296605 18972 ts_tablet_manager.cc:1367] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.004s
I20240501 08:12:34.297550 18972 raft_consensus.cc:348] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.297852 18972 raft_consensus.cc:374] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.298857 13213 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:34.299566 18972 raft_consensus.cc:729] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.299906 18972 consensus_queue.cc:260] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.306988 18972 ts_tablet_manager.cc:1398] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 08:12:34.314230 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 74b3abba8f5d47a5ac65c5cce94459d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f6261e1204844c2a213b9c132786654]), partition=
I20240501 08:12:34.314636 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b3abba8f5d47a5ac65c5cce94459d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:34.316744 19000 raft_consensus.cc:482] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:34.316949 19000 raft_consensus.cc:504] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.318755 19000 leader_election.cc:288] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:34.319124 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa10a6f53f7f48cf9410c80e1056439d" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:34.319327 12974 raft_consensus.cc:2454] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:34.319377 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa10a6f53f7f48cf9410c80e1056439d" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:34.319658 16025 raft_consensus.cc:2454] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:34.320163 15826 leader_election.cc:302] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:34.321578 18749 raft_consensus.cc:482] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:34.321729 18749 raft_consensus.cc:504] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.322467 18999 raft_consensus.cc:2781] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:34.322564 18749 leader_election.cc:288] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:34.322731 18999 raft_consensus.cc:482] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:34.322871 18999 raft_consensus.cc:3037] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.323431 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8ec0d302644d378c54e4871f85159a" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:34.323632 15891 raft_consensus.cc:2454] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:34.324512 18973 ts_tablet_manager.cc:1398] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.063s user 0.000s sys 0.003s
I20240501 08:12:34.326169 15957 leader_election.cc:302] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:34.326373 19011 tablet_bootstrap.cc:492] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:34.326475 18749 raft_consensus.cc:2781] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:34.326596 18749 raft_consensus.cc:482] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:34.326701 18749 raft_consensus.cc:3037] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.328375 19011 tablet_bootstrap.cc:654] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.329525 18749 raft_consensus.cc:504] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.331189 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8ec0d302644d378c54e4871f85159a" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:34.331377 16153 raft_consensus.cc:3037] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.331411 18749 leader_election.cc:288] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:34.332044 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8ec0d302644d378c54e4871f85159a" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:34.332257 15891 raft_consensus.cc:3037] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.334151 16153 raft_consensus.cc:2454] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:34.334867 15733 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:12:34.334977 15891 raft_consensus.cc:2454] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:34.335534 15957 leader_election.cc:302] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:34.336468 18202 raft_consensus.cc:2781] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:34.336627 18202 raft_consensus.cc:686] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:34.336882 18202 consensus_queue.cc:237] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.338727 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c8ec0d302644d378c54e4871f85159a" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:34.338977 16154 raft_consensus.cc:2362] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1. Re-sending same reply.
I20240501 08:12:34.347898 19000 raft_consensus.cc:482] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:34.348335 19011 tablet_bootstrap.cc:492] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:34.348615 19011 ts_tablet_manager.cc:1367] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20240501 08:12:34.348589 19000 raft_consensus.cc:504] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.349586 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f305a6e24ab1436b895db037f5d72870" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:34.349807 13364 raft_consensus.cc:2454] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:34.350303 15825 leader_election.cc:302] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters:
I20240501 08:12:34.350647 18439 raft_consensus.cc:2781] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:34.350740 18439 raft_consensus.cc:482] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:34.350837 18439 raft_consensus.cc:3037] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.352432 18999 raft_consensus.cc:504] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.353614 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa10a6f53f7f48cf9410c80e1056439d" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:34.353611 18439 raft_consensus.cc:504] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.353790 12974 raft_consensus.cc:3037] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.353690 19011 raft_consensus.cc:348] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:34.353938 19011 raft_consensus.cc:374] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.354055 19011 raft_consensus.cc:729] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.354197 18439 leader_election.cc:288] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:34.354938 19011 consensus_queue.cc:260] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:34.355409 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f305a6e24ab1436b895db037f5d72870" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:34.355588 13364 raft_consensus.cc:3037] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.357607 12974 raft_consensus.cc:2454] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:34.357921 18999 leader_election.cc:288] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:34.358431 19011 ts_tablet_manager.cc:1398] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240501 08:12:34.358704 13364 raft_consensus.cc:2454] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:34.359231 15826 leader_election.cc:302] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:34.359627 18999 raft_consensus.cc:2781] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:34.359922 18999 raft_consensus.cc:686] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:34.360841 15825 leader_election.cc:302] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters:
I20240501 08:12:34.361553 18439 raft_consensus.cc:2781] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:34.361060 18999 consensus_queue.cc:237] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.361757 18439 raft_consensus.cc:686] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:34.362093 18439 consensus_queue.cc:237] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:34.362761 19000 leader_election.cc:288] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:34.376262 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa10a6f53f7f48cf9410c80e1056439d" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:34.376520 16024 raft_consensus.cc:3037] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.379343 16024 raft_consensus.cc:2454] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:34.387723 12827 catalog_manager.cc:5484] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:34.417807 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f305a6e24ab1436b895db037f5d72870" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:34.418073 16154 raft_consensus.cc:3037] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.421257 16154 raft_consensus.cc:2454] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:34.433948 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f305a6e24ab1436b895db037f5d72870" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:34.434453 16154 raft_consensus.cc:2362] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1. Re-sending same reply.
I20240501 08:12:34.455776 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 74b3abba8f5d47a5ac65c5cce94459d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f6261e1204844c2a213b9c132786654]), partition=
I20240501 08:12:34.456329 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74b3abba8f5d47a5ac65c5cce94459d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:34.463850 12954 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:34.466202 12832 catalog_manager.cc:5484] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:34.466792 12832 catalog_manager.cc:5484] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:34.484136 13085 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:34.491061 19019 tablet_bootstrap.cc:492] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:34.494670 19019 tablet_bootstrap.cc:654] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:34.507426 19019 tablet_bootstrap.cc:492] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:34.507889 19019 ts_tablet_manager.cc:1367] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.010s
I20240501 08:12:34.509419 19019 raft_consensus.cc:348] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:34.509886 19019 raft_consensus.cc:374] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:34.510030 19019 raft_consensus.cc:729] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:34.510423 19019 consensus_queue.cc:260] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:34.529942 19019 ts_tablet_manager.cc:1398] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240501 08:12:34.624353 12974 consensus_queue.cc:237] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:34.626581 16025 raft_consensus.cc:1261] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:34.626672 13235 raft_consensus.cc:1261] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:34.627238 18970 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:34.627542 18714 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:34.630398 18970 raft_consensus.cc:2932] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:34.631240 13235 raft_consensus.cc:2932] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:34.638624 12832 catalog_manager.cc:5484] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:34.642717 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet 427bc3b61a874f83bc27d9be24732e00 with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:34.643015 18992 tablet_replica.cc:330] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:34.643280 18992 raft_consensus.cc:2229] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:34.643568 18992 raft_consensus.cc:2258] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:34.644706 18992 ts_tablet_manager.cc:1875] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:34.652976 18992 ts_tablet_manager.cc:1888] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:34.652535 16025 raft_consensus.cc:2932] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:34.653256 18992 log.cc:1196] T 427bc3b61a874f83bc27d9be24732e00 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/427bc3b61a874f83bc27d9be24732e00
I20240501 08:12:34.654119 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 427bc3b61a874f83bc27d9be24732e00 (table rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]) successfully deleted
I20240501 08:12:34.724090 18202 consensus_queue.cc:1035] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.735064 18202 consensus_queue.cc:1035] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.759029 18970 raft_consensus.cc:482] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:34.759213 18970 raft_consensus.cc:504] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:34.760113 18970 leader_election.cc:288] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:34.761240 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3abba8f5d47a5ac65c5cce94459d5" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:34.761502 13104 raft_consensus.cc:2454] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:34.762135 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3abba8f5d47a5ac65c5cce94459d5" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:34.762387 15754 raft_consensus.cc:2454] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:34.763696 12909 leader_election.cc:302] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:34.764206 18714 raft_consensus.cc:2781] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:34.764321 18714 raft_consensus.cc:482] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:34.764400 18714 raft_consensus.cc:3037] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.767139 18714 raft_consensus.cc:504] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:34.767750 18714 leader_election.cc:288] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:34.768251 18999 consensus_queue.cc:1035] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.769171 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3abba8f5d47a5ac65c5cce94459d5" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:34.769301 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74b3abba8f5d47a5ac65c5cce94459d5" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:34.769493 15754 raft_consensus.cc:3037] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.769526 13104 raft_consensus.cc:3037] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:34.774403 13104 raft_consensus.cc:2454] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:34.775113 15754 raft_consensus.cc:2454] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 1.
I20240501 08:12:34.775161 12909 leader_election.cc:302] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:34.775466 18970 raft_consensus.cc:2781] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:34.775629 18970 raft_consensus.cc:686] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:12:34.775899 18970 consensus_queue.cc:237] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:34.798794 18439 consensus_queue.cc:1035] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.802477 12832 catalog_manager.cc:5484] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:34.911518 18999 consensus_queue.cc:1035] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:34.950579 18999 consensus_queue.cc:1035] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:35.004755 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:35.013221 15870 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:12:35.013533 12824 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:35.024653 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 2cca8e4588be4791800840a638bf0372 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:35.025233 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca8e4588be4791800840a638bf0372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.025662 13103 raft_consensus.cc:1261] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:35.026818 18714 consensus_queue.cc:1035] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:35.028797 15754 raft_consensus.cc:1261] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:35.028935 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 2cca8e4588be4791800840a638bf0372 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:35.029368 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca8e4588be4791800840a638bf0372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.029755 19075 consensus_queue.cc:1035] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:35.033051 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 2cca8e4588be4791800840a638bf0372 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:35.033535 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca8e4588be4791800840a638bf0372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.043323 13084 tablet_service.cc:1467] Processing CreateTablet for tablet b000ec8b190644f2883e5f824463b762 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:35.043809 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000ec8b190644f2883e5f824463b762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.044495 16137 tablet_service.cc:1467] Processing CreateTablet for tablet b000ec8b190644f2883e5f824463b762 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:35.046005 16005 tablet_service.cc:1467] Processing CreateTablet for tablet ba08236c71df4c8190f6b7644ed944ed (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:35.046413 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba08236c71df4c8190f6b7644ed944ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.048692 13213 tablet_service.cc:1467] Processing CreateTablet for tablet ba08236c71df4c8190f6b7644ed944ed (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:35.048743 16004 tablet_service.cc:1467] Processing CreateTablet for tablet b000ec8b190644f2883e5f824463b762 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:35.049153 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000ec8b190644f2883e5f824463b762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.049834 15736 tablet_service.cc:1467] Processing CreateTablet for tablet ba08236c71df4c8190f6b7644ed944ed (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:35.050338 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba08236c71df4c8190f6b7644ed944ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.052369 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000ec8b190644f2883e5f824463b762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.052567 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba08236c71df4c8190f6b7644ed944ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.081872 13214 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:35.085764 19119 tablet_bootstrap.cc:492] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:35.089548 19119 tablet_bootstrap.cc:654] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.098692 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:35.112008 19122 tablet_bootstrap.cc:492] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:35.114379 19122 tablet_bootstrap.cc:654] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.125548 12954 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:12:35.130152 19117 tablet_bootstrap.cc:492] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:35.134356 19119 tablet_bootstrap.cc:492] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:35.134408 19117 tablet_bootstrap.cc:654] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.134546 19119 ts_tablet_manager.cc:1367] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.004s
I20240501 08:12:35.137240 19120 tablet_bootstrap.cc:492] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:35.138800 19120 tablet_bootstrap.cc:654] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.140687 16005 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:12:35.148775 19119 raft_consensus.cc:348] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.150606 19119 raft_consensus.cc:374] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.151244 19119 raft_consensus.cc:729] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.151695 19119 consensus_queue.cc:260] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.154024 19119 ts_tablet_manager.cc:1398] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.019s user 0.001s sys 0.002s
I20240501 08:12:35.158155 19119 tablet_bootstrap.cc:492] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:35.157877 19122 tablet_bootstrap.cc:492] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:35.159771 19122 ts_tablet_manager.cc:1367] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.002s
I20240501 08:12:35.160776 19122 raft_consensus.cc:348] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.161080 19122 raft_consensus.cc:374] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.161183 19122 raft_consensus.cc:729] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.161477 19122 consensus_queue.cc:260] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.162271 19122 ts_tablet_manager.cc:1398] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:35.163975 19119 tablet_bootstrap.cc:654] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.172006 19133 tablet.cc:1722] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:35.173142 19133 tablet.cc:1620] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:35.173954 19119 tablet_bootstrap.cc:492] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:35.181300 19118 tablet_bootstrap.cc:492] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:35.186681 19130 tablet.cc:1722] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:35.192399 19130 tablet.cc:1620] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:35.213485 12824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=7f6261e1204844c2a213b9c132786654] alter complete (version 1)
I20240501 08:12:35.219045 12824 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet 74b3abba8f5d47a5ac65c5cce94459d5 (table rebalancer_dynamic_table [id=7f6261e1204844c2a213b9c132786654]). Expected version 1 got 0
I20240501 08:12:35.220571 19118 tablet_bootstrap.cc:654] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.223850 12824 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 74b3abba8f5d47a5ac65c5cce94459d5 (table rebalancer_dynamic_table [id=7f6261e1204844c2a213b9c132786654]). Expected version 1 got 0
I20240501 08:12:35.227279 19119 ts_tablet_manager.cc:1367] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.002s
I20240501 08:12:35.230031 19120 tablet_bootstrap.cc:492] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:35.230177 19120 ts_tablet_manager.cc:1367] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.000s
I20240501 08:12:35.230983 19120 raft_consensus.cc:348] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.231185 19120 raft_consensus.cc:374] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.231240 19120 raft_consensus.cc:729] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.231491 19120 consensus_queue.cc:260] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.232024 19120 ts_tablet_manager.cc:1398] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:35.232972 19119 raft_consensus.cc:348] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.233757 13344 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:35.234679 19119 raft_consensus.cc:374] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.241431 19118 tablet_bootstrap.cc:492] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:35.241650 19118 ts_tablet_manager.cc:1367] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.000s
I20240501 08:12:35.244203 19118 raft_consensus.cc:348] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.244462 19118 raft_consensus.cc:374] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.244580 19118 raft_consensus.cc:729] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.248373 19119 raft_consensus.cc:729] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.245494 19118 consensus_queue.cc:260] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.249609 19118 ts_tablet_manager.cc:1398] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 08:12:35.251322 12824 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet 74b3abba8f5d47a5ac65c5cce94459d5 (table rebalancer_dynamic_table [id=7f6261e1204844c2a213b9c132786654]). Expected version 1 got 0
I20240501 08:12:35.251888 19118 tablet_bootstrap.cc:492] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:35.255326 19117 tablet_bootstrap.cc:492] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:35.255600 19117 ts_tablet_manager.cc:1367] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.002s
I20240501 08:12:35.256435 19117 raft_consensus.cc:348] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.258255 19117 raft_consensus.cc:374] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.258417 19117 raft_consensus.cc:729] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.258759 19117 consensus_queue.cc:260] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.258924 19118 tablet_bootstrap.cc:654] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.260838 18815 raft_consensus.cc:482] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:35.261009 18815 raft_consensus.cc:504] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.261775 18815 leader_election.cc:288] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:35.262916 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000ec8b190644f2883e5f824463b762" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:35.263032 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000ec8b190644f2883e5f824463b762" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:35.263221 13103 raft_consensus.cc:2454] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
W20240501 08:12:35.263525 16092 leader_election.cc:341] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:35.263733 16092 leader_election.cc:302] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters: 5dd722c770d34d6c9154a87917117161
I20240501 08:12:35.264444 19117 ts_tablet_manager.cc:1398] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240501 08:12:35.266247 19117 tablet_bootstrap.cc:492] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:35.268445 19117 tablet_bootstrap.cc:654] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.248185 19136 tablet.cc:1722] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:35.278126 19119 consensus_queue.cc:260] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.279482 19117 tablet_bootstrap.cc:492] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:35.279722 19117 ts_tablet_manager.cc:1367] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240501 08:12:35.280232 19136 tablet.cc:1620] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:35.282269 19119 ts_tablet_manager.cc:1398] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.055s user 0.001s sys 0.001s
I20240501 08:12:35.283473 19117 raft_consensus.cc:348] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.283833 19117 raft_consensus.cc:374] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.283978 19117 raft_consensus.cc:729] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.284314 19117 consensus_queue.cc:260] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.286895 19117 ts_tablet_manager.cc:1398] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:12:35.297106 13084 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:12:35.301177 19139 raft_consensus.cc:2781] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:35.301316 19139 raft_consensus.cc:482] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:35.301437 19139 raft_consensus.cc:3037] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.317831 19117 tablet_bootstrap.cc:492] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:35.321771 19139 raft_consensus.cc:504] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.323498 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000ec8b190644f2883e5f824463b762" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:35.323714 13103 raft_consensus.cc:3037] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.329254 13103 raft_consensus.cc:2454] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:35.329967 16092 leader_election.cc:302] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:35.330467 18815 raft_consensus.cc:2781] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:35.331858 12825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:35.335486 18815 raft_consensus.cc:686] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:35.335906 18815 consensus_queue.cc:237] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.338697 19117 tablet_bootstrap.cc:654] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.344548 19130 tablet.cc:1722] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:35.345067 19130 tablet.cc:1620] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:35.345846 19133 tablet.cc:1722] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:35.346344 19133 tablet.cc:1620] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:35.347909 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b000ec8b190644f2883e5f824463b762" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:35.348114 16025 raft_consensus.cc:3037] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.350823 16025 raft_consensus.cc:2454] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:35.354398 19136 tablet.cc:1722] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:35.354843 19136 tablet.cc:1620] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:35.355123 12825 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=7f6261e1204844c2a213b9c132786654] alter complete (version 2)
I20240501 08:12:35.382838 19139 leader_election.cc:288] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:35.388748 15736 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:12:35.397363 19141 raft_consensus.cc:482] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:35.399924 19117 tablet_bootstrap.cc:492] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:35.400221 19117 ts_tablet_manager.cc:1367] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.082s user 0.006s sys 0.002s
I20240501 08:12:35.397562 19141 raft_consensus.cc:504] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.401546 19141 leader_election.cc:288] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:35.401774 19117 raft_consensus.cc:348] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.402072 19117 raft_consensus.cc:374] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.402240 19117 raft_consensus.cc:729] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.402523 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca8e4588be4791800840a638bf0372" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:35.402585 19117 consensus_queue.cc:260] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.403255 19117 ts_tablet_manager.cc:1398] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:12:35.404404 19118 tablet_bootstrap.cc:492] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:35.404668 19118 ts_tablet_manager.cc:1367] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.153s user 0.031s sys 0.021s
I20240501 08:12:35.405612 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca8e4588be4791800840a638bf0372" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:35.405854 16025 raft_consensus.cc:2454] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:35.406248 15691 leader_election.cc:302] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:35.406559 19141 raft_consensus.cc:2781] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:35.406682 19141 raft_consensus.cc:482] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:35.406766 19141 raft_consensus.cc:3037] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:12:35.407476 15689 leader_election.cc:341] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:35.408998 19118 raft_consensus.cc:348] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.409392 19118 raft_consensus.cc:374] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.409523 19118 raft_consensus.cc:729] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.409724 19141 raft_consensus.cc:504] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.409874 19118 consensus_queue.cc:260] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.410712 19118 ts_tablet_manager.cc:1398] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:35.411099 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca8e4588be4791800840a638bf0372" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:35.411279 16154 raft_consensus.cc:3037] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.413560 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cca8e4588be4791800840a638bf0372" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:35.413808 16025 raft_consensus.cc:3037] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.414654 16154 raft_consensus.cc:2454] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:35.415050 15689 leader_election.cc:302] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:35.415352 19010 raft_consensus.cc:2781] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:35.416042 19141 leader_election.cc:288] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:35.422151 16025 raft_consensus.cc:2454] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:35.432686 19010 raft_consensus.cc:686] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:35.433250 19010 consensus_queue.cc:237] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:35.450691 12824 catalog_manager.cc:5484] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:35.461288 12833 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:35.477469 19130 tablet.cc:1722] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:35.478268 19133 tablet.cc:1722] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:35.479974 12833 catalog_manager.cc:5484] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:35.489564 12824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=7f6261e1204844c2a213b9c132786654] alter complete (version 3)
I20240501 08:12:35.494590 19010 raft_consensus.cc:482] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:35.494838 19010 raft_consensus.cc:504] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.495607 19010 leader_election.cc:288] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:35.496052 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba08236c71df4c8190f6b7644ed944ed" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:35.496336 16025 raft_consensus.cc:2454] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:35.496269 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba08236c71df4c8190f6b7644ed944ed" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:35.496500 13235 raft_consensus.cc:2454] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:35.496845 15691 leader_election.cc:302] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:35.497424 19010 raft_consensus.cc:2781] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:35.497560 19010 raft_consensus.cc:482] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:35.497717 19010 raft_consensus.cc:3037] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.501353 19010 raft_consensus.cc:504] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.502768 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba08236c71df4c8190f6b7644ed944ed" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:35.503062 16025 raft_consensus.cc:3037] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.506882 16025 raft_consensus.cc:2454] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:35.507390 15691 leader_election.cc:302] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:35.507933 19146 raft_consensus.cc:2781] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:35.508069 19146 raft_consensus.cc:686] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:35.508376 19146 consensus_queue.cc:237] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.511442 19136 tablet.cc:1722] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:35.518509 19010 leader_election.cc:288] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:35.520565 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba08236c71df4c8190f6b7644ed944ed" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:35.520769 13235 raft_consensus.cc:3037] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.523651 13235 raft_consensus.cc:2454] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:35.531307 12833 catalog_manager.cc:5484] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:35.591974 12833 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:35.592191 12833 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:35.596757 12833 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 74b3abba8f5d47a5ac65c5cce94459d5
I20240501 08:12:35.597867 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b3abba8f5d47a5ac65c5cce94459d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:35 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:35.597924 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b3abba8f5d47a5ac65c5cce94459d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:35 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:35.597900 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 74b3abba8f5d47a5ac65c5cce94459d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:35 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:35.600664 19157 tablet_replica.cc:330] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:35.600976 19157 raft_consensus.cc:2229] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:35.601263 19157 raft_consensus.cc:2258] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:35.602142 19157 ts_tablet_manager.cc:1875] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:35.608999 19157 ts_tablet_manager.cc:1888] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:35.609138 19157 log.cc:1196] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/74b3abba8f5d47a5ac65c5cce94459d5
I20240501 08:12:35.609596 19157 ts_tablet_manager.cc:1909] T 74b3abba8f5d47a5ac65c5cce94459d5 P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:35.610661 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 74b3abba8f5d47a5ac65c5cce94459d5 (table rebalancer_dynamic_table_new_name [id=7f6261e1204844c2a213b9c132786654]) successfully deleted
I20240501 08:12:35.612085 19156 tablet_replica.cc:330] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:35.612344 19156 raft_consensus.cc:2229] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:35.612815 19156 raft_consensus.cc:2258] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:35.613451 19158 tablet_replica.cc:330] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:35.613637 19158 raft_consensus.cc:2229] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:35.613925 19158 raft_consensus.cc:2258] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:35.614939 19156 ts_tablet_manager.cc:1875] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:35.617005 19158 ts_tablet_manager.cc:1875] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:35.620790 19156 ts_tablet_manager.cc:1888] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:35.620913 19156 log.cc:1196] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/74b3abba8f5d47a5ac65c5cce94459d5
I20240501 08:12:35.621297 19156 ts_tablet_manager.cc:1909] T 74b3abba8f5d47a5ac65c5cce94459d5 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:35.622385 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 74b3abba8f5d47a5ac65c5cce94459d5 (table rebalancer_dynamic_table_new_name [id=7f6261e1204844c2a213b9c132786654]) successfully deleted
I20240501 08:12:35.623034 19158 ts_tablet_manager.cc:1888] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:35.623126 19158 log.cc:1196] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/74b3abba8f5d47a5ac65c5cce94459d5
I20240501 08:12:35.623385 19158 ts_tablet_manager.cc:1909] T 74b3abba8f5d47a5ac65c5cce94459d5 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:35.624325 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 74b3abba8f5d47a5ac65c5cce94459d5 (table rebalancer_dynamic_table_new_name [id=7f6261e1204844c2a213b9c132786654]) successfully deleted
I20240501 08:12:35.632565 16154 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c2c8548c31343ff9426fe7eda194703"
dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38850
I20240501 08:12:35.632856 16154 raft_consensus.cc:595] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:35.701786 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:35.719548 15870 tablet_service.cc:1467] Processing CreateTablet for tablet fb094cf16c7c436987ad4337c208591c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9581843ca26469ebd354587721ff4c0]), partition=
I20240501 08:12:35.720139 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb094cf16c7c436987ad4337c208591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.720593 15736 tablet_service.cc:1467] Processing CreateTablet for tablet fb094cf16c7c436987ad4337c208591c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9581843ca26469ebd354587721ff4c0]), partition=
I20240501 08:12:35.720993 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb094cf16c7c436987ad4337c208591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.725013 12954 tablet_service.cc:1467] Processing CreateTablet for tablet fb094cf16c7c436987ad4337c208591c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9581843ca26469ebd354587721ff4c0]), partition=
I20240501 08:12:35.725741 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb094cf16c7c436987ad4337c208591c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:35.743878 19139 raft_consensus.cc:979] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: : Instructing follower e3fefd4dca2d41678187047f06c10187 to start an election
I20240501 08:12:35.744094 19139 raft_consensus.cc:1067] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Signalling peer e3fefd4dca2d41678187047f06c10187 to start an election
I20240501 08:12:35.745069 13103 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c2c8548c31343ff9426fe7eda194703"
dest_uuid: "e3fefd4dca2d41678187047f06c10187"
from {username='slave'} at 127.6.56.72:42629
I20240501 08:12:35.745319 13103 raft_consensus.cc:482] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:35.745429 13103 raft_consensus.cc:3037] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:35.745674 19184 tablet_bootstrap.cc:492] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:35.747275 19184 tablet_bootstrap.cc:654] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.747756 19119 tablet_bootstrap.cc:492] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:35.749722 19119 tablet_bootstrap.cc:654] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.751963 13103 raft_consensus.cc:504] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.754652 19184 tablet_bootstrap.cc:492] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:35.755151 19184 ts_tablet_manager.cc:1367] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240501 08:12:35.755132 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2c8548c31343ff9426fe7eda194703" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:35.755352 15754 raft_consensus.cc:3037] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:35.755292 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2c8548c31343ff9426fe7eda194703" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:35.755565 13235 raft_consensus.cc:3037] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:35.755887 19184 raft_consensus.cc:348] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.756073 19184 raft_consensus.cc:374] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.756142 19184 raft_consensus.cc:729] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.756373 19184 consensus_queue.cc:260] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.756914 19184 ts_tablet_manager.cc:1398] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:35.758495 19144 consensus_queue.cc:1035] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:35.758896 13235 raft_consensus.cc:2454] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 2.
I20240501 08:12:35.761976 15754 raft_consensus.cc:2454] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 2.
I20240501 08:12:35.767596 13041 leader_election.cc:302] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 982f49fc87f149439cda0de2261b6bfd, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:35.767877 19177 raft_consensus.cc:2781] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:35.768008 19177 raft_consensus.cc:686] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 2 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:35.768201 19119 tablet_bootstrap.cc:492] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:35.768262 19177 consensus_queue.cc:237] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.768448 19119 ts_tablet_manager.cc:1367] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20240501 08:12:35.768621 19144 consensus_queue.cc:1035] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:35.769722 19186 tablet_bootstrap.cc:492] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:35.769618 19119 raft_consensus.cc:348] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.769872 19119 raft_consensus.cc:374] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.770104 19119 raft_consensus.cc:729] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.770426 19119 consensus_queue.cc:260] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.771557 19119 ts_tablet_manager.cc:1398] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:35.773665 13103 leader_election.cc:288] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:35.802116 19186 tablet_bootstrap.cc:654] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:35.802060 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c2c8548c31343ff9426fe7eda194703" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:35.802417 16154 raft_consensus.cc:3032] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:35.802529 16154 raft_consensus.cc:729] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:35.802932 16154 consensus_queue.cc:260] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.803592 16154 raft_consensus.cc:3037] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:35.814249 19186 tablet_bootstrap.cc:492] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:35.814448 19186 ts_tablet_manager.cc:1367] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.008s
I20240501 08:12:35.815647 19186 raft_consensus.cc:348] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.815874 19186 raft_consensus.cc:374] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:35.816058 19186 raft_consensus.cc:729] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:35.816479 19186 consensus_queue.cc:260] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.820017 19186 ts_tablet_manager.cc:1398] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:12:35.814626 16154 raft_consensus.cc:2454] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 2.
I20240501 08:12:35.813334 12833 catalog_manager.cc:5484] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 1 to 2, leader changed from 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 2 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:35.826483 19010 consensus_queue.cc:1035] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:35.842391 19146 consensus_queue.cc:1035] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:35.852483 18999 raft_consensus.cc:482] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:35.852708 18999 raft_consensus.cc:504] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.853770 18999 leader_election.cc:288] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:35.854060 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb094cf16c7c436987ad4337c208591c" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:35.854279 12974 raft_consensus.cc:2454] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:35.854838 15826 leader_election.cc:302] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:35.855190 18999 raft_consensus.cc:2781] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:35.855297 18999 raft_consensus.cc:482] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:35.855415 18999 raft_consensus.cc:3037] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.857019 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb094cf16c7c436987ad4337c208591c" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:35.857263 15754 raft_consensus.cc:2454] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:35.858608 18999 raft_consensus.cc:504] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.859632 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb094cf16c7c436987ad4337c208591c" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:35.859834 12974 raft_consensus.cc:3037] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.862804 12974 raft_consensus.cc:2454] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:35.863364 15826 leader_election.cc:302] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:35.863713 19018 raft_consensus.cc:2781] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:35.871932 18999 leader_election.cc:288] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:35.936951 19018 raft_consensus.cc:686] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:35.937538 19018 consensus_queue.cc:237] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:35.950827 12824 catalog_manager.cc:5484] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:35.952780 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb094cf16c7c436987ad4337c208591c" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:35.953014 15754 raft_consensus.cc:3037] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:35.955914 15754 raft_consensus.cc:2454] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:36.037530 19010 consensus_queue.cc:1035] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:36.039961 19146 consensus_queue.cc:1035] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:36.062358 12831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:36.075917 13213 tablet_service.cc:1467] Processing CreateTablet for tablet d38e22c9e39d43bc818eede00882ff8d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:36.075994 16137 tablet_service.cc:1467] Processing CreateTablet for tablet d38e22c9e39d43bc818eede00882ff8d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:36.076666 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38e22c9e39d43bc818eede00882ff8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.078260 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38e22c9e39d43bc818eede00882ff8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.083611 16005 tablet_service.cc:1467] Processing CreateTablet for tablet d38e22c9e39d43bc818eede00882ff8d (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:36.084084 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d38e22c9e39d43bc818eede00882ff8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.086175 19218 tablet_bootstrap.cc:492] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:36.089349 12831 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:36.095322 19218 tablet_bootstrap.cc:654] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.099802 12974 raft_consensus.cc:1261] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.099849 15754 raft_consensus.cc:1261] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:36.100608 19205 consensus_queue.cc:1035] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:36.102110 19203 consensus_queue.cc:1035] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:36.110210 19218 tablet_bootstrap.cc:492] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:36.110496 19218 ts_tablet_manager.cc:1367] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.007s
I20240501 08:12:36.111618 19218 raft_consensus.cc:348] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.111970 19218 raft_consensus.cc:374] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.112118 19218 raft_consensus.cc:729] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.112442 19218 consensus_queue.cc:260] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.114470 19218 ts_tablet_manager.cc:1398] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:36.118119 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 691f5a8c1d334f6ba6464c9596db49ed (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:36.118763 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691f5a8c1d334f6ba6464c9596db49ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.122382 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 691f5a8c1d334f6ba6464c9596db49ed (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:36.122953 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691f5a8c1d334f6ba6464c9596db49ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.124737 13343 tablet_service.cc:1467] Processing CreateTablet for tablet b8a350eade1c41e4b4c29e487374552f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:36.125149 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a350eade1c41e4b4c29e487374552f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.128916 19224 tablet_bootstrap.cc:492] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:36.135913 19224 tablet_bootstrap.cc:654] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.144976 15870 tablet_service.cc:1467] Processing CreateTablet for tablet b8a350eade1c41e4b4c29e487374552f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:36.145509 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a350eade1c41e4b4c29e487374552f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.145987 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 691f5a8c1d334f6ba6464c9596db49ed (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:36.147162 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691f5a8c1d334f6ba6464c9596db49ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.155583 15733 tablet_service.cc:1467] Processing CreateTablet for tablet b8a350eade1c41e4b4c29e487374552f (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:36.156033 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a350eade1c41e4b4c29e487374552f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.163244 19119 tablet_bootstrap.cc:492] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:36.164573 13232 raft_consensus.cc:1261] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:12:36.166071 19119 tablet_bootstrap.cc:654] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.166795 19177 consensus_queue.cc:1035] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:36.169378 12824 catalog_manager.cc:5576] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839) does not have the latest schema for tablet fb094cf16c7c436987ad4337c208591c (table rebalancer_dynamic_table [id=a9581843ca26469ebd354587721ff4c0]). Expected version 1 got 0
I20240501 08:12:36.176976 19119 tablet_bootstrap.cc:492] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:36.177408 19119 ts_tablet_manager.cc:1367] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.009s
I20240501 08:12:36.178730 19119 raft_consensus.cc:348] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.179152 19119 raft_consensus.cc:374] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.179347 19119 raft_consensus.cc:729] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.179783 19119 consensus_queue.cc:260] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.180859 19119 ts_tablet_manager.cc:1398] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:12:36.225600 19186 tablet_bootstrap.cc:492] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:36.228168 19186 tablet_bootstrap.cc:654] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.245741 19230 tablet.cc:1722] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:36.246587 19230 tablet.cc:1620] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:36.248533 15755 raft_consensus.cc:1261] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:12:36.249692 19177 consensus_queue.cc:1035] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:36.251190 12832 catalog_manager.cc:5576] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839) does not have the latest schema for tablet fb094cf16c7c436987ad4337c208591c (table rebalancer_dynamic_table [id=a9581843ca26469ebd354587721ff4c0]). Expected version 1 got 0
I20240501 08:12:36.257151 12831 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet fb094cf16c7c436987ad4337c208591c (table rebalancer_dynamic_table [id=a9581843ca26469ebd354587721ff4c0]). Expected version 1 got 0
I20240501 08:12:36.259918 16153 raft_consensus.cc:1261] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:12:36.260452 19177 consensus_queue.cc:1035] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:36.265836 19186 tablet_bootstrap.cc:492] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:36.266109 19186 ts_tablet_manager.cc:1367] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.007s
I20240501 08:12:36.267261 19186 raft_consensus.cc:348] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.267706 19186 raft_consensus.cc:374] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.267839 19186 raft_consensus.cc:729] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.268218 19186 consensus_queue.cc:260] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.269225 19186 ts_tablet_manager.cc:1398] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:36.295305 19119 tablet_bootstrap.cc:492] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:36.297008 19119 tablet_bootstrap.cc:654] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.316140 13343 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:36.342486 19144 raft_consensus.cc:482] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:36.342762 19144 raft_consensus.cc:504] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.343715 19144 leader_election.cc:288] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:36.344736 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38e22c9e39d43bc818eede00882ff8d" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:36.345160 13231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38e22c9e39d43bc818eede00882ff8d" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
W20240501 08:12:36.345746 16093 leader_election.cc:341] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:12:36.346015 16092 leader_election.cc:341] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:36.346176 16092 leader_election.cc:302] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:36.346688 19144 raft_consensus.cc:2726] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:36.353478 19203 raft_consensus.cc:482] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:36.353695 19203 raft_consensus.cc:504] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.354480 19203 leader_election.cc:288] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:36.355264 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a350eade1c41e4b4c29e487374552f" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
W20240501 08:12:36.355998 15825 leader_election.cc:341] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:36.356544 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a350eade1c41e4b4c29e487374552f" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:36.356781 15755 raft_consensus.cc:2454] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:36.361053 12824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=a9581843ca26469ebd354587721ff4c0] alter complete (version 1)
I20240501 08:12:36.363065 15825 leader_election.cc:302] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters: 4ecb724cb4764a53988fde00407db4c0
I20240501 08:12:36.363549 19203 raft_consensus.cc:2781] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:36.363655 19203 raft_consensus.cc:482] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:36.363765 19203 raft_consensus.cc:3037] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:36.366757 19203 raft_consensus.cc:504] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.373099 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a350eade1c41e4b4c29e487374552f" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:36.373370 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a350eade1c41e4b4c29e487374552f" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:36.373600 15755 raft_consensus.cc:3037] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:12:36.374666 15825 leader_election.cc:341] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:36.372051 19203 leader_election.cc:288] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:36.376791 19228 tablet.cc:1722] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:36.377491 19228 tablet.cc:1620] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:36.378095 15755 raft_consensus.cc:2454] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:36.378795 15825 leader_election.cc:302] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters: 4ecb724cb4764a53988fde00407db4c0
I20240501 08:12:36.379940 19203 raft_consensus.cc:2781] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:36.380244 19203 raft_consensus.cc:686] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:36.380661 19203 consensus_queue.cc:237] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.387171 19256 tablet_bootstrap.cc:492] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:36.389359 19256 tablet_bootstrap.cc:654] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.390836 19257 tablet_bootstrap.cc:492] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:36.393595 19257 tablet_bootstrap.cc:654] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.406433 19119 tablet_bootstrap.cc:492] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:36.406649 19119 ts_tablet_manager.cc:1367] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.006s
I20240501 08:12:36.407598 19119 raft_consensus.cc:348] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.407881 19119 raft_consensus.cc:374] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.407999 19119 raft_consensus.cc:729] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.416368 13213 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:36.408228 19119 consensus_queue.cc:260] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.427040 19119 ts_tablet_manager.cc:1398] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240501 08:12:36.428762 12954 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:36.432932 19229 tablet.cc:1722] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:36.433796 19229 tablet.cc:1620] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:36.442552 12825 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet fb094cf16c7c436987ad4337c208591c (table rebalancer_dynamic_table [id=a9581843ca26469ebd354587721ff4c0]). Expected version 1 got 0
I20240501 08:12:36.445690 16005 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:12:36.462812 19257 tablet_bootstrap.cc:492] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:36.463080 19257 ts_tablet_manager.cc:1367] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.072s user 0.000s sys 0.004s
I20240501 08:12:36.464010 19257 raft_consensus.cc:348] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.464412 19257 raft_consensus.cc:374] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.464542 19257 raft_consensus.cc:729] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.464900 19257 consensus_queue.cc:260] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.471262 12825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:36.475804 16137 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:36.478423 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:36.485899 19256 tablet_bootstrap.cc:492] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:36.486196 19256 ts_tablet_manager.cc:1367] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.099s user 0.000s sys 0.004s
I20240501 08:12:36.489677 19256 raft_consensus.cc:348] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.490407 19256 raft_consensus.cc:374] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.490545 19256 raft_consensus.cc:729] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.490836 19256 consensus_queue.cc:260] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.492399 19230 tablet.cc:1722] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:36.492944 19230 tablet.cc:1620] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:36.492866 19228 tablet.cc:1722] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:36.491461 19256 ts_tablet_manager.cc:1398] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:12:36.493981 19257 ts_tablet_manager.cc:1398] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.031s user 0.000s sys 0.002s
I20240501 08:12:36.494379 19256 tablet_bootstrap.cc:492] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:36.494803 19228 tablet.cc:1620] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:36.498612 19256 tablet_bootstrap.cc:654] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
W20240501 08:12:36.501937 13038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 105133us
I20240501 08:12:36.506866 15870 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:36.507001 19224 tablet_bootstrap.cc:492] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:36.507167 19224 ts_tablet_manager.cc:1367] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.378s user 0.008s sys 0.024s
I20240501 08:12:36.510135 19224 raft_consensus.cc:348] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.520995 12831 catalog_manager.cc:5484] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:36.531657 19229 tablet.cc:1722] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:36.532164 19229 tablet.cc:1620] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:36.536646 19256 tablet_bootstrap.cc:492] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:36.538537 19256 ts_tablet_manager.cc:1367] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.001s
I20240501 08:12:36.539376 19256 raft_consensus.cc:348] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.539675 19256 raft_consensus.cc:374] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.539772 19256 raft_consensus.cc:729] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.540125 19256 consensus_queue.cc:260] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.542907 19256 ts_tablet_manager.cc:1398] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:36.546537 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=a9581843ca26469ebd354587721ff4c0] alter complete (version 2)
I20240501 08:12:36.547789 19224 raft_consensus.cc:374] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.547981 19224 raft_consensus.cc:729] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.548324 19224 consensus_queue.cc:260] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.555395 13084 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:12:36.578200 19224 ts_tablet_manager.cc:1398] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.069s user 0.000s sys 0.002s
I20240501 08:12:36.586869 19224 tablet_bootstrap.cc:492] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:36.594453 19010 raft_consensus.cc:482] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:36.594674 19010 raft_consensus.cc:504] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.595427 19010 leader_election.cc:288] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:36.596086 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691f5a8c1d334f6ba6464c9596db49ed" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:36.596607 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691f5a8c1d334f6ba6464c9596db49ed" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:36.596805 13363 raft_consensus.cc:2454] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 0.
I20240501 08:12:36.597429 15691 leader_election.cc:302] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters:
W20240501 08:12:36.597730 15691 leader_election.cc:341] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:36.598575 19010 raft_consensus.cc:2781] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:36.598680 19010 raft_consensus.cc:482] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:36.598783 19010 raft_consensus.cc:3037] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:36.606648 19010 raft_consensus.cc:504] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.607975 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691f5a8c1d334f6ba6464c9596db49ed" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:36.608186 13363 raft_consensus.cc:3037] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:36.610828 13363 raft_consensus.cc:2454] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 1.
I20240501 08:12:36.611498 15691 leader_election.cc:302] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:36.611841 19146 raft_consensus.cc:2781] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:36.614280 19146 raft_consensus.cc:686] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:12:36.614619 19146 consensus_queue.cc:237] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.621663 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691f5a8c1d334f6ba6464c9596db49ed" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
W20240501 08:12:36.622359 15691 leader_election.cc:341] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:36.625634 12825 catalog_manager.cc:5484] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:36.625913 19010 leader_election.cc:288] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:36.639931 19224 tablet_bootstrap.cc:654] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.652544 12825 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:36.665241 19228 tablet.cc:1722] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:36.665431 19230 tablet.cc:1722] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240501 08:12:36.669987 15691 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 44687.6us
I20240501 08:12:36.672626 19229 tablet.cc:1722] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:36.682332 12825 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=a9581843ca26469ebd354587721ff4c0] alter complete (version 3)
I20240501 08:12:36.693730 18202 raft_consensus.cc:482] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:36.693940 18202 raft_consensus.cc:504] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.697728 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38e22c9e39d43bc818eede00882ff8d" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:36.697949 13235 raft_consensus.cc:2454] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:36.698555 18202 leader_election.cc:288] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:36.698546 19224 tablet_bootstrap.cc:492] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:36.699031 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38e22c9e39d43bc818eede00882ff8d" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:36.699297 16158 raft_consensus.cc:2454] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:36.700037 15960 leader_election.cc:302] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:36.700606 18202 raft_consensus.cc:2781] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:36.701257 18202 raft_consensus.cc:482] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:36.701370 18202 raft_consensus.cc:3037] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:36.705763 18202 raft_consensus.cc:504] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.707158 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38e22c9e39d43bc818eede00882ff8d" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:36.707439 16158 raft_consensus.cc:3037] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:36.710552 16158 raft_consensus.cc:2454] T d38e22c9e39d43bc818eede00882ff8d P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:36.714937 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d38e22c9e39d43bc818eede00882ff8d" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:36.715263 13235 raft_consensus.cc:3037] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:36.720036 13235 raft_consensus.cc:2454] T d38e22c9e39d43bc818eede00882ff8d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:36.720814 15957 leader_election.cc:302] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:36.721450 18749 raft_consensus.cc:2781] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:36.721599 18749 raft_consensus.cc:686] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:36.725306 19224 ts_tablet_manager.cc:1367] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.112s user 0.000s sys 0.005s
I20240501 08:12:36.723718 18749 consensus_queue.cc:237] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.728812 18202 leader_election.cc:288] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:36.743382 19224 raft_consensus.cc:348] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.743724 19224 raft_consensus.cc:374] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.743816 19224 raft_consensus.cc:729] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.744165 19224 consensus_queue.cc:260] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:36.744825 19224 ts_tablet_manager.cc:1398] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240501 08:12:36.763549 12825 catalog_manager.cc:5484] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:36.786283 12825 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:36.786629 12825 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:36.790055 12825 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet fb094cf16c7c436987ad4337c208591c
I20240501 08:12:36.790824 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet fb094cf16c7c436987ad4337c208591c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:36 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:36.790843 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet fb094cf16c7c436987ad4337c208591c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:36 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:36.790952 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet fb094cf16c7c436987ad4337c208591c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:36 UTC) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:36.791427 19282 tablet_replica.cc:330] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:36.791688 19282 raft_consensus.cc:2229] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:36.792282 19282 raft_consensus.cc:2258] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:36.792826 19284 tablet_replica.cc:330] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:36.793277 19284 raft_consensus.cc:2229] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:36.793390 19282 ts_tablet_manager.cc:1875] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:36.794099 19284 raft_consensus.cc:2258] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:36.795435 19284 ts_tablet_manager.cc:1875] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:36.799408 19283 tablet_replica.cc:330] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:36.799674 19283 raft_consensus.cc:2229] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:36.799870 19283 raft_consensus.cc:2258] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:36.800963 19282 ts_tablet_manager.cc:1888] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:36.801182 19282 log.cc:1196] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/fb094cf16c7c436987ad4337c208591c
I20240501 08:12:36.802028 19283 ts_tablet_manager.cc:1875] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:36.802258 19282 ts_tablet_manager.cc:1909] T fb094cf16c7c436987ad4337c208591c P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:36.803429 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet fb094cf16c7c436987ad4337c208591c (table rebalancer_dynamic_table_new_name [id=a9581843ca26469ebd354587721ff4c0]) successfully deleted
I20240501 08:12:36.805102 19284 ts_tablet_manager.cc:1888] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:36.805308 19284 log.cc:1196] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/fb094cf16c7c436987ad4337c208591c
I20240501 08:12:36.805809 19284 ts_tablet_manager.cc:1909] T fb094cf16c7c436987ad4337c208591c P 982f49fc87f149439cda0de2261b6bfd: Deleting consensus metadata
I20240501 08:12:36.806918 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet fb094cf16c7c436987ad4337c208591c (table rebalancer_dynamic_table_new_name [id=a9581843ca26469ebd354587721ff4c0]) successfully deleted
I20240501 08:12:36.811167 19283 ts_tablet_manager.cc:1888] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:36.811302 19283 log.cc:1196] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/fb094cf16c7c436987ad4337c208591c
I20240501 08:12:36.811699 19283 ts_tablet_manager.cc:1909] T fb094cf16c7c436987ad4337c208591c P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:36.812652 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet fb094cf16c7c436987ad4337c208591c (table rebalancer_dynamic_table_new_name [id=a9581843ca26469ebd354587721ff4c0]) successfully deleted
I20240501 08:12:36.839723 19124 raft_consensus.cc:482] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:36.839910 19124 raft_consensus.cc:504] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.840672 19124 leader_election.cc:288] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:36.841413 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a350eade1c41e4b4c29e487374552f" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:36.841565 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8a350eade1c41e4b4c29e487374552f" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:36.841707 15755 raft_consensus.cc:2379] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4ecb724cb4764a53988fde00407db4c0 in current term 1: Already voted for candidate 4d516916eeac468db514eeb928f06190 in this term.
I20240501 08:12:36.842273 13298 leader_election.cc:302] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0; no voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:36.842716 19124 raft_consensus.cc:3037] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:36.845135 19124 raft_consensus.cc:2726] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:36.854477 13104 consensus_queue.cc:237] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:36.856225 13235 raft_consensus.cc:1261] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:36.856804 19266 consensus_queue.cc:1035] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.856971 15755 raft_consensus.cc:1261] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:12:36.857522 19266 consensus_queue.cc:1035] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:36.859186 19268 raft_consensus.cc:2932] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:36.859932 13235 raft_consensus.cc:2932] T 6c2c8548c31343ff9426fe7eda194703 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:36.861910 15755 raft_consensus.cc:2932] T 6c2c8548c31343ff9426fe7eda194703 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:36.866989 12825 catalog_manager.cc:5484] T 6c2c8548c31343ff9426fe7eda194703 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 2 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:36.872789 16137 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c2c8548c31343ff9426fe7eda194703 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:36.873458 19308 tablet_replica.cc:330] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:36.873682 19308 raft_consensus.cc:2229] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:36.873946 19308 raft_consensus.cc:2258] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:36.874859 19308 ts_tablet_manager.cc:1875] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:36.881122 19308 ts_tablet_manager.cc:1888] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 08:12:36.881325 19308 log.cc:1196] T 6c2c8548c31343ff9426fe7eda194703 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/6c2c8548c31343ff9426fe7eda194703
I20240501 08:12:36.882198 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 6c2c8548c31343ff9426fe7eda194703 (table rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]) successfully deleted
I20240501 08:12:36.893191 12824 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:36.908812 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 69d7bf152f714633a0e08f6312f71ca0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=91475c1ad89d48478ad3d7c7a9d9a469]), partition=
I20240501 08:12:36.909106 13344 tablet_service.cc:1467] Processing CreateTablet for tablet 69d7bf152f714633a0e08f6312f71ca0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=91475c1ad89d48478ad3d7c7a9d9a469]), partition=
I20240501 08:12:36.909163 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 69d7bf152f714633a0e08f6312f71ca0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=91475c1ad89d48478ad3d7c7a9d9a469]), partition=
I20240501 08:12:36.909253 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d7bf152f714633a0e08f6312f71ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.909482 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d7bf152f714633a0e08f6312f71ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.909497 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d7bf152f714633a0e08f6312f71ca0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:36.909554 19018 consensus_queue.cc:1035] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:36.916994 19224 tablet_bootstrap.cc:492] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:36.918733 19224 tablet_bootstrap.cc:654] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.921980 19256 tablet_bootstrap.cc:492] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:36.923318 19256 tablet_bootstrap.cc:654] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:36.935402 19018 consensus_queue.cc:1035] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:36.948014 19224 tablet_bootstrap.cc:492] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:36.948184 19224 ts_tablet_manager.cc:1367] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.005s
I20240501 08:12:36.951071 19224 raft_consensus.cc:348] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.951309 19224 raft_consensus.cc:374] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.951395 19224 raft_consensus.cc:729] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.951633 19224 consensus_queue.cc:260] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.955704 19224 ts_tablet_manager.cc:1398] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:12:36.977946 19256 tablet_bootstrap.cc:492] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:36.979022 19315 tablet_bootstrap.cc:492] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:36.978199 19256 ts_tablet_manager.cc:1367] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.056s user 0.006s sys 0.006s
I20240501 08:12:36.983672 19256 raft_consensus.cc:348] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.989601 19256 raft_consensus.cc:374] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:36.989708 19256 raft_consensus.cc:729] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:36.989925 19256 consensus_queue.cc:260] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:36.990556 19256 ts_tablet_manager.cc:1398] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:12:36.991286 19315 tablet_bootstrap.cc:654] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.002184 19315 tablet_bootstrap.cc:492] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:37.002341 19315 ts_tablet_manager.cc:1367] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.008s
I20240501 08:12:37.002909 19315 raft_consensus.cc:348] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:37.003089 19315 raft_consensus.cc:374] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.003129 19315 raft_consensus.cc:729] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.003332 19315 consensus_queue.cc:260] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:37.003809 19315 ts_tablet_manager.cc:1398] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 08:12:37.030126 18749 raft_consensus.cc:482] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:37.030334 18749 raft_consensus.cc:504] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.030968 18749 leader_election.cc:288] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:37.031602 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691f5a8c1d334f6ba6464c9596db49ed" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:37.031723 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "691f5a8c1d334f6ba6464c9596db49ed" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:37.031965 13364 raft_consensus.cc:2379] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5dd722c770d34d6c9154a87917117161 in current term 1: Already voted for candidate 982f49fc87f149439cda0de2261b6bfd in this term.
I20240501 08:12:37.032689 15959 leader_election.cc:302] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:37.033294 19280 raft_consensus.cc:3037] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.039762 19280 raft_consensus.cc:2726] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:37.041839 19275 consensus_queue.cc:1035] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:37.049646 19010 consensus_queue.cc:1035] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:37.063702 19280 raft_consensus.cc:482] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:37.064045 19280 raft_consensus.cc:504] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:37.064953 19280 leader_election.cc:288] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:37.065685 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d7bf152f714633a0e08f6312f71ca0" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:37.065888 13364 raft_consensus.cc:2454] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:37.066812 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d7bf152f714633a0e08f6312f71ca0" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:37.067045 13104 raft_consensus.cc:2454] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:37.068387 15959 leader_election.cc:302] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:37.068749 19279 raft_consensus.cc:2781] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:37.068841 19279 raft_consensus.cc:482] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:37.068939 19279 raft_consensus.cc:3037] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.072278 19279 raft_consensus.cc:504] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:37.074523 19279 leader_election.cc:288] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:37.076040 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d7bf152f714633a0e08f6312f71ca0" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:37.076261 13104 raft_consensus.cc:3037] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.081043 13104 raft_consensus.cc:2454] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:37.082198 15959 leader_election.cc:302] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:37.082579 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d7bf152f714633a0e08f6312f71ca0" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:37.082762 13364 raft_consensus.cc:3037] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.085430 13364 raft_consensus.cc:2454] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:37.089115 19279 raft_consensus.cc:2781] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:37.089452 19279 raft_consensus.cc:686] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:37.089751 19279 consensus_queue.cc:237] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:37.112820 12824 catalog_manager.cc:5484] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:37.166602 19279 consensus_queue.cc:1035] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:37.175109 19279 consensus_queue.cc:1035] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:37.222579 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:37.231772 13364 raft_consensus.cc:1261] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:37.231772 13104 raft_consensus.cc:1261] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:37.232460 18749 consensus_queue.cc:1035] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:37.232846 19280 consensus_queue.cc:1035] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:37.262104 12831 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet 69d7bf152f714633a0e08f6312f71ca0 (table rebalancer_dynamic_table [id=91475c1ad89d48478ad3d7c7a9d9a469]). Expected version 1 got 0
I20240501 08:12:37.268880 12831 catalog_manager.cc:5576] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509) does not have the latest schema for tablet 69d7bf152f714633a0e08f6312f71ca0 (table rebalancer_dynamic_table [id=91475c1ad89d48478ad3d7c7a9d9a469]). Expected version 1 got 0
W20240501 08:12:37.270840 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.66:35509, user_credentials={real_user=slave}} blocked reactor thread for 57007.2us
I20240501 08:12:37.272850 19367 tablet.cc:1722] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:37.273782 19367 tablet.cc:1620] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:37.277127 19370 tablet.cc:1722] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:37.280396 19373 tablet.cc:1722] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:37.280982 19373 tablet.cc:1620] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:37.282868 12833 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet 69d7bf152f714633a0e08f6312f71ca0 (table rebalancer_dynamic_table [id=91475c1ad89d48478ad3d7c7a9d9a469]). Expected version 1 got 0
I20240501 08:12:37.292963 12824 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=91475c1ad89d48478ad3d7c7a9d9a469] alter complete (version 1)
I20240501 08:12:37.294355 19370 tablet.cc:1620] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:37.318697 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:37.337841 12954 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba04c9b0d0a4933a123dbeb5b50145e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:37.338357 12954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba04c9b0d0a4933a123dbeb5b50145e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.338840 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba04c9b0d0a4933a123dbeb5b50145e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:37.339296 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba04c9b0d0a4933a123dbeb5b50145e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.340716 15736 tablet_service.cc:1467] Processing CreateTablet for tablet b2e7c8c1795d4c6d997c75e63dee6760 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:37.341171 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e7c8c1795d4c6d997c75e63dee6760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.341845 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 21719a43fa6845f8a3a5337dbe770f60 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:37.342244 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21719a43fa6845f8a3a5337dbe770f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.344707 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 21719a43fa6845f8a3a5337dbe770f60 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:37.345166 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21719a43fa6845f8a3a5337dbe770f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.348778 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 21719a43fa6845f8a3a5337dbe770f60 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:37.349274 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21719a43fa6845f8a3a5337dbe770f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.352223 19224 tablet_bootstrap.cc:492] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:37.352938 16004 tablet_service.cc:1467] Processing CreateTablet for tablet b2e7c8c1795d4c6d997c75e63dee6760 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:37.353384 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e7c8c1795d4c6d997c75e63dee6760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.354197 19224 tablet_bootstrap.cc:654] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.356141 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba04c9b0d0a4933a123dbeb5b50145e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:37.356598 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba04c9b0d0a4933a123dbeb5b50145e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.358928 19224 tablet_bootstrap.cc:492] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:37.359097 19224 ts_tablet_manager.cc:1367] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:12:37.359736 19224 raft_consensus.cc:348] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.359911 19224 raft_consensus.cc:374] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.359977 19224 raft_consensus.cc:729] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.360195 19224 consensus_queue.cc:260] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.360756 19224 ts_tablet_manager.cc:1398] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:37.361150 19224 tablet_bootstrap.cc:492] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:37.362133 12953 tablet_service.cc:1467] Processing CreateTablet for tablet b2e7c8c1795d4c6d997c75e63dee6760 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:37.362607 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e7c8c1795d4c6d997c75e63dee6760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.363551 19224 tablet_bootstrap.cc:654] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.367425 19224 tablet_bootstrap.cc:492] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:37.367664 19224 ts_tablet_manager.cc:1367] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240501 08:12:37.368975 19224 raft_consensus.cc:348] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.369341 19224 raft_consensus.cc:374] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.369549 19224 raft_consensus.cc:729] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.369849 19224 consensus_queue.cc:260] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.371996 19224 ts_tablet_manager.cc:1398] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:37.372408 19224 tablet_bootstrap.cc:492] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:37.374423 19380 tablet_bootstrap.cc:492] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:37.378945 19380 tablet_bootstrap.cc:654] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.380012 19224 tablet_bootstrap.cc:654] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.381749 19380 tablet_bootstrap.cc:492] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:37.381961 19380 ts_tablet_manager.cc:1367] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 08:12:37.382917 19380 raft_consensus.cc:348] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.383073 19380 raft_consensus.cc:374] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.383147 19380 raft_consensus.cc:729] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.383356 19380 consensus_queue.cc:260] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.383417 19224 tablet_bootstrap.cc:492] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:37.383607 19224 ts_tablet_manager.cc:1367] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240501 08:12:37.384503 19224 raft_consensus.cc:348] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.384753 19224 raft_consensus.cc:374] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.384842 19224 raft_consensus.cc:729] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.385182 19224 consensus_queue.cc:260] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.386651 19380 ts_tablet_manager.cc:1398] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:12:37.386952 19380 tablet_bootstrap.cc:492] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:37.387089 19224 ts_tablet_manager.cc:1398] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:37.388903 19380 tablet_bootstrap.cc:654] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.397691 19315 tablet_bootstrap.cc:492] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:37.399681 19315 tablet_bootstrap.cc:654] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.402879 19384 tablet_bootstrap.cc:492] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:37.406003 19384 tablet_bootstrap.cc:654] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.408051 12831 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:37.426076 19384 tablet_bootstrap.cc:492] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:37.426319 19384 ts_tablet_manager.cc:1367] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.006s
I20240501 08:12:37.427368 19384 raft_consensus.cc:348] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.427435 19367 tablet.cc:1722] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:37.427662 19384 raft_consensus.cc:374] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.427796 19384 raft_consensus.cc:729] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.428068 19367 tablet.cc:1620] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:37.428089 19384 consensus_queue.cc:260] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.429526 19384 ts_tablet_manager.cc:1398] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:12:37.429819 19384 tablet_bootstrap.cc:492] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:37.432726 19384 tablet_bootstrap.cc:654] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.438220 19384 tablet_bootstrap.cc:492] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:37.438547 19384 ts_tablet_manager.cc:1367] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20240501 08:12:37.440104 19370 tablet.cc:1722] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:37.440614 19370 tablet.cc:1620] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:37.441817 19384 raft_consensus.cc:348] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.442234 19384 raft_consensus.cc:374] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.442374 19384 raft_consensus.cc:729] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.442827 19384 consensus_queue.cc:260] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.445758 19384 ts_tablet_manager.cc:1398] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240501 08:12:37.446899 19373 tablet.cc:1722] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:37.447451 19373 tablet.cc:1620] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:37.469453 12831 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=91475c1ad89d48478ad3d7c7a9d9a469] alter complete (version 2)
I20240501 08:12:37.490377 19385 tablet_bootstrap.cc:492] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:37.491058 19360 raft_consensus.cc:482] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:37.491283 19360 raft_consensus.cc:504] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.492156 19360 leader_election.cc:288] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:37.494246 19380 tablet_bootstrap.cc:492] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:37.494436 19380 ts_tablet_manager.cc:1367] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.108s user 0.003s sys 0.010s
I20240501 08:12:37.497412 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba04c9b0d0a4933a123dbeb5b50145e" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:37.497673 16024 raft_consensus.cc:2454] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 0.
I20240501 08:12:37.499655 16092 leader_election.cc:302] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:37.499827 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba04c9b0d0a4933a123dbeb5b50145e" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:37.500142 19368 raft_consensus.cc:2781] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:37.500263 19368 raft_consensus.cc:482] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:37.500653 19368 raft_consensus.cc:3037] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.503576 13344 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:37.504855 19380 raft_consensus.cc:348] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.505163 19380 raft_consensus.cc:374] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.505236 19380 raft_consensus.cc:729] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
W20240501 08:12:37.505348 16093 leader_election.cc:341] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:37.505488 19380 consensus_queue.cc:260] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.505020 19368 raft_consensus.cc:504] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.506837 19385 tablet_bootstrap.cc:654] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.506904 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba04c9b0d0a4933a123dbeb5b50145e" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:37.507079 12974 raft_consensus.cc:3037] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.511994 19368 leader_election.cc:288] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:37.515192 15870 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240501 08:12:37.517030 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ba04c9b0d0a4933a123dbeb5b50145e" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:37.517256 19315 tablet_bootstrap.cc:492] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:37.517273 16025 raft_consensus.cc:3037] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.517446 19315 ts_tablet_manager.cc:1367] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.120s user 0.000s sys 0.013s
I20240501 08:12:37.518507 19315 raft_consensus.cc:348] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.518817 19315 raft_consensus.cc:374] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.518880 19315 raft_consensus.cc:729] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.519227 19315 consensus_queue.cc:260] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.519990 16025 raft_consensus.cc:2454] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:37.520422 19380 ts_tablet_manager.cc:1398] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.026s user 0.000s sys 0.003s
I20240501 08:12:37.520731 19315 ts_tablet_manager.cc:1398] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:12:37.528520 16092 leader_election.cc:302] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:37.531400 12974 raft_consensus.cc:2454] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 1.
I20240501 08:12:37.532603 19144 raft_consensus.cc:2781] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:37.532752 19144 raft_consensus.cc:686] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:12:37.533066 19144 consensus_queue.cc:237] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.567549 19266 raft_consensus.cc:482] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:37.567716 19266 raft_consensus.cc:504] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.568408 19266 leader_election.cc:288] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:37.575162 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21719a43fa6845f8a3a5337dbe770f60" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:37.575454 16025 raft_consensus.cc:2454] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:37.576843 13040 leader_election.cc:302] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:37.577266 19266 raft_consensus.cc:2781] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:37.577369 19266 raft_consensus.cc:482] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:37.577446 19266 raft_consensus.cc:3037] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.581054 19266 raft_consensus.cc:504] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.582854 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21719a43fa6845f8a3a5337dbe770f60" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:37.583091 16024 raft_consensus.cc:3037] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.588881 16024 raft_consensus.cc:2454] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:37.595378 13040 leader_election.cc:302] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:37.595682 19401 raft_consensus.cc:2781] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:37.595846 19401 raft_consensus.cc:686] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:37.596319 19401 consensus_queue.cc:237] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.598591 12831 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:37.615072 12830 catalog_manager.cc:5484] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:37.616147 19373 tablet.cc:1722] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:37.616060 19367 tablet.cc:1722] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:37.621436 19370 tablet.cc:1722] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:37.623378 19266 leader_election.cc:288] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:37.635134 19368 raft_consensus.cc:482] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:37.635398 19368 raft_consensus.cc:504] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:37.636143 19368 leader_election.cc:288] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:37.636981 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21719a43fa6845f8a3a5337dbe770f60" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:37.645746 19279 raft_consensus.cc:482] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:37.646198 19279 raft_consensus.cc:504] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.648623 19279 leader_election.cc:288] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:37.648931 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e7c8c1795d4c6d997c75e63dee6760" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:37.649233 12973 raft_consensus.cc:2454] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:37.649801 15960 leader_election.cc:302] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:37.651041 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e7c8c1795d4c6d997c75e63dee6760" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:37.651645 18749 raft_consensus.cc:2781] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:37.651818 18749 raft_consensus.cc:482] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:37.651935 18749 raft_consensus.cc:3037] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.655336 18749 raft_consensus.cc:504] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.656828 18749 leader_election.cc:288] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:37.658774 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e7c8c1795d4c6d997c75e63dee6760" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:37.659077 12973 raft_consensus.cc:3037] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.660665 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=91475c1ad89d48478ad3d7c7a9d9a469] alter complete (version 3)
W20240501 08:12:37.674022 15959 leader_election.cc:341] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:37.676555 12973 raft_consensus.cc:2454] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:37.677080 15960 leader_election.cc:302] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:37.677495 19279 raft_consensus.cc:2781] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:37.677714 19279 raft_consensus.cc:686] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:37.678018 19279 consensus_queue.cc:237] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.679883 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e7c8c1795d4c6d997c75e63dee6760" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
W20240501 08:12:37.681624 15959 leader_election.cc:341] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:37.711732 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21719a43fa6845f8a3a5337dbe770f60" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:37.712157 16025 raft_consensus.cc:2379] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6cc32f6e2d49477f9a23ce6e444039fb in current term 1: Already voted for candidate e3fefd4dca2d41678187047f06c10187 in this term.
I20240501 08:12:37.713872 16092 leader_election.cc:302] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187
I20240501 08:12:37.715101 19400 raft_consensus.cc:3037] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.720538 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21719a43fa6845f8a3a5337dbe770f60" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:37.721228 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21719a43fa6845f8a3a5337dbe770f60" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:37.728778 19385 tablet_bootstrap.cc:492] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:37.729043 19385 ts_tablet_manager.cc:1367] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.239s user 0.049s sys 0.129s
I20240501 08:12:37.730015 19385 raft_consensus.cc:348] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.730353 19385 raft_consensus.cc:374] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.730516 19385 raft_consensus.cc:729] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.730793 19385 consensus_queue.cc:260] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.731520 19385 ts_tablet_manager.cc:1398] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:37.745702 12833 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:37.746125 12833 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:37.750279 12833 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 69d7bf152f714633a0e08f6312f71ca0
I20240501 08:12:37.751742 16004 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d7bf152f714633a0e08f6312f71ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:37 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:37.752147 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d7bf152f714633a0e08f6312f71ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:37 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:37.760699 19408 tablet_replica.cc:330] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:37.760699 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d7bf152f714633a0e08f6312f71ca0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:37 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:37.760989 19408 raft_consensus.cc:2229] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:37.774829 19408 raft_consensus.cc:2258] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:37.776126 19408 ts_tablet_manager.cc:1875] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:37.795233 19410 tablet_replica.cc:330] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:37.795477 19410 raft_consensus.cc:2229] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:37.795692 19410 raft_consensus.cc:2258] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:37.796955 12831 catalog_manager.cc:5484] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:37.809197 12830 catalog_manager.cc:5484] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:37.809576 19408 ts_tablet_manager.cc:1888] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:37.809736 19408 log.cc:1196] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/69d7bf152f714633a0e08f6312f71ca0
I20240501 08:12:37.811105 19408 ts_tablet_manager.cc:1909] T 69d7bf152f714633a0e08f6312f71ca0 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:37.817636 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 69d7bf152f714633a0e08f6312f71ca0 (table rebalancer_dynamic_table_new_name [id=91475c1ad89d48478ad3d7c7a9d9a469]) successfully deleted
I20240501 08:12:37.820185 19409 tablet_replica.cc:330] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:37.820511 19409 raft_consensus.cc:2229] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:37.821146 19410 ts_tablet_manager.cc:1875] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:37.822825 19409 raft_consensus.cc:2258] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:37.824023 19409 ts_tablet_manager.cc:1875] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:37.831972 19410 ts_tablet_manager.cc:1888] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:37.832242 19410 log.cc:1196] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/69d7bf152f714633a0e08f6312f71ca0
I20240501 08:12:37.832859 19410 ts_tablet_manager.cc:1909] T 69d7bf152f714633a0e08f6312f71ca0 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:37.834475 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 69d7bf152f714633a0e08f6312f71ca0 (table rebalancer_dynamic_table_new_name [id=91475c1ad89d48478ad3d7c7a9d9a469]) successfully deleted
I20240501 08:12:37.839401 19400 raft_consensus.cc:2726] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:37.840621 16154 raft_consensus.cc:2454] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:37.845069 16137 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:37.862094 19409 ts_tablet_manager.cc:1888] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:37.862221 19409 log.cc:1196] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/69d7bf152f714633a0e08f6312f71ca0
I20240501 08:12:37.862623 19409 ts_tablet_manager.cc:1909] T 69d7bf152f714633a0e08f6312f71ca0 P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:37.867648 16004 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:37.871338 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 69d7bf152f714633a0e08f6312f71ca0 (table rebalancer_dynamic_table_new_name [id=91475c1ad89d48478ad3d7c7a9d9a469]) successfully deleted
I20240501 08:12:37.874673 12831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:37.909464 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 33ea19edb3b54fc48962f89532b4f36d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78891ec772a2428091464473d9c4b9d5]), partition=
I20240501 08:12:37.909947 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ea19edb3b54fc48962f89532b4f36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.910238 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 33ea19edb3b54fc48962f89532b4f36d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78891ec772a2428091464473d9c4b9d5]), partition=
I20240501 08:12:37.910823 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ea19edb3b54fc48962f89532b4f36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.911669 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 33ea19edb3b54fc48962f89532b4f36d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=78891ec772a2428091464473d9c4b9d5]), partition=
I20240501 08:12:37.912147 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33ea19edb3b54fc48962f89532b4f36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:37.929029 19384 tablet_bootstrap.cc:492] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:37.932368 19277 raft_consensus.cc:482] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:37.932595 19277 raft_consensus.cc:504] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:37.933552 19277 leader_election.cc:288] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:37.933967 19315 tablet_bootstrap.cc:492] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:37.938038 12973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e7c8c1795d4c6d997c75e63dee6760" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:37.938361 12973 raft_consensus.cc:2379] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 982f49fc87f149439cda0de2261b6bfd in current term 1: Already voted for candidate 5dd722c770d34d6c9154a87917117161 in this term.
I20240501 08:12:37.939123 19315 tablet_bootstrap.cc:654] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.941040 16025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e7c8c1795d4c6d997c75e63dee6760" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:37.942359 15691 leader_election.cc:302] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:37.943053 19277 raft_consensus.cc:3037] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:37.947068 19277 raft_consensus.cc:2726] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:37.954171 19413 tablet_bootstrap.cc:492] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:37.959390 19400 consensus_queue.cc:1035] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20240501 08:12:37.960173 19384 tablet_bootstrap.cc:654] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.967965 19315 tablet_bootstrap.cc:492] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:37.968351 19315 ts_tablet_manager.cc:1367] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.002s
I20240501 08:12:37.969110 19315 raft_consensus.cc:348] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:37.969403 19315 raft_consensus.cc:374] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.969509 19315 raft_consensus.cc:729] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.969897 19315 consensus_queue.cc:260] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:37.973533 19315 ts_tablet_manager.cc:1398] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 08:12:37.978693 19413 tablet_bootstrap.cc:654] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:37.988014 19384 tablet_bootstrap.cc:492] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:37.988606 19384 ts_tablet_manager.cc:1367] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.006s
I20240501 08:12:37.989634 19384 raft_consensus.cc:348] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:37.989979 19384 raft_consensus.cc:374] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:37.990161 19384 raft_consensus.cc:729] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:37.990274 13213 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:37.990483 19384 consensus_queue.cc:260] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:37.991218 19384 ts_tablet_manager.cc:1398] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:38.016569 19401 raft_consensus.cc:482] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:38.016777 19401 raft_consensus.cc:504] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:38.017673 19401 leader_election.cc:288] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:38.018687 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ea19edb3b54fc48962f89532b4f36d" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:38.019505 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ea19edb3b54fc48962f89532b4f36d" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:38.019804 16154 raft_consensus.cc:2454] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
W20240501 08:12:38.020329 13038 leader_election.cc:341] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:38.021284 13038 leader_election.cc:302] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters: 4d516916eeac468db514eeb928f06190
I20240501 08:12:38.021770 19401 raft_consensus.cc:2781] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:38.021843 19401 raft_consensus.cc:482] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:38.021901 19401 raft_consensus.cc:3037] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.026038 19401 raft_consensus.cc:504] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:38.027199 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ea19edb3b54fc48962f89532b4f36d" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:38.027424 16154 raft_consensus.cc:3037] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.029740 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33ea19edb3b54fc48962f89532b4f36d" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:38.030264 16154 raft_consensus.cc:2454] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
W20240501 08:12:38.030380 13038 leader_election.cc:341] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:38.030772 13038 leader_election.cc:302] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb, e3fefd4dca2d41678187047f06c10187; no voters: 4d516916eeac468db514eeb928f06190
I20240501 08:12:38.031041 19266 raft_consensus.cc:2781] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:38.031162 19266 raft_consensus.cc:686] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:38.031433 19266 consensus_queue.cc:237] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:38.036396 19401 leader_election.cc:288] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:38.053843 12831 catalog_manager.cc:5484] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:38.067107 19368 consensus_queue.cc:1035] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:38.085402 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:38.087550 13084 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:12:38.113154 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:38.113624 19413 tablet_bootstrap.cc:492] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:38.113860 19413 ts_tablet_manager.cc:1367] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.160s user 0.004s sys 0.013s
I20240501 08:12:38.114753 19413 raft_consensus.cc:348] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:38.115113 19413 raft_consensus.cc:374] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.115309 19413 raft_consensus.cc:729] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.130182 19413 consensus_queue.cc:260] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:38.133926 19413 ts_tablet_manager.cc:1398] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.020s user 0.000s sys 0.001s
I20240501 08:12:38.151763 19266 consensus_queue.cc:1035] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:38.175757 19268 consensus_queue.cc:1035] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:38.182561 19404 consensus_queue.cc:1035] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:38.189519 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:38.219992 15891 raft_consensus.cc:3037] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.223620 15891 raft_consensus.cc:1261] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:38.225605 19401 consensus_queue.cc:1035] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:38.227809 16157 raft_consensus.cc:1261] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:38.228605 19401 consensus_queue.cc:1035] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:38.246949 19279 consensus_queue.cc:1035] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:38.302261 12830 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet 33ea19edb3b54fc48962f89532b4f36d (table rebalancer_dynamic_table [id=78891ec772a2428091464473d9c4b9d5]). Expected version 1 got 0
I20240501 08:12:38.319747 12831 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 33ea19edb3b54fc48962f89532b4f36d (table rebalancer_dynamic_table [id=78891ec772a2428091464473d9c4b9d5]). Expected version 1 got 0
I20240501 08:12:38.349819 19441 tablet.cc:1722] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:38.352767 19441 tablet.cc:1620] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:38.353588 12831 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet 33ea19edb3b54fc48962f89532b4f36d (table rebalancer_dynamic_table [id=78891ec772a2428091464473d9c4b9d5]). Expected version 1 got 0
I20240501 08:12:38.355947 19440 tablet.cc:1722] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:38.357357 12831 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:38.357113 19440 tablet.cc:1620] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:38.360671 19442 tablet.cc:1722] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:38.361419 19442 tablet.cc:1620] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:38.363420 12833 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 33ea19edb3b54fc48962f89532b4f36d (table rebalancer_dynamic_table [id=78891ec772a2428091464473d9c4b9d5]). Expected version 1 got 0
I20240501 08:12:38.369439 13213 tablet_service.cc:1467] Processing CreateTablet for tablet ca6339e48a2f4b9e94e32e05c7bf343b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:38.369477 16004 tablet_service.cc:1467] Processing CreateTablet for tablet ca6339e48a2f4b9e94e32e05c7bf343b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:38.369915 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6339e48a2f4b9e94e32e05c7bf343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.369954 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6339e48a2f4b9e94e32e05c7bf343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.373811 12953 tablet_service.cc:1467] Processing CreateTablet for tablet ca6339e48a2f4b9e94e32e05c7bf343b (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:38.374266 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6339e48a2f4b9e94e32e05c7bf343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.376996 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 80ef1222e2324309a009a657ca704795 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:38.377127 13214 tablet_service.cc:1467] Processing CreateTablet for tablet 80ef1222e2324309a009a657ca704795 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:38.377528 13214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ef1222e2324309a009a657ca704795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.382056 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 80ef1222e2324309a009a657ca704795 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:38.382514 15733 tablet_service.cc:1467] Processing CreateTablet for tablet d09d4bf8b82941199313f03bc1abeab5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:38.383452 13344 tablet_service.cc:1467] Processing CreateTablet for tablet d09d4bf8b82941199313f03bc1abeab5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:38.383832 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09d4bf8b82941199313f03bc1abeab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.385426 13215 tablet_service.cc:1467] Processing CreateTablet for tablet d09d4bf8b82941199313f03bc1abeab5 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:38.385809 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09d4bf8b82941199313f03bc1abeab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.389313 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ef1222e2324309a009a657ca704795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.382536 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ef1222e2324309a009a657ca704795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.396569 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d09d4bf8b82941199313f03bc1abeab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:38.419134 12830 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=78891ec772a2428091464473d9c4b9d5] alter complete (version 1)
I20240501 08:12:38.423916 19448 tablet_bootstrap.cc:492] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:38.429230 19448 tablet_bootstrap.cc:654] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:38.431874 19447 tablet_bootstrap.cc:492] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:38.436158 19445 tablet_bootstrap.cc:492] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:38.437213 19447 tablet_bootstrap.cc:654] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:38.439157 19448 tablet_bootstrap.cc:492] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:38.439399 19448 ts_tablet_manager.cc:1367] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240501 08:12:38.440588 19448 raft_consensus.cc:348] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:38.440922 19448 raft_consensus.cc:374] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.441073 19448 raft_consensus.cc:729] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.441371 19448 consensus_queue.cc:260] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:38.442170 19445 tablet_bootstrap.cc:654] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:38.463781 19448 ts_tablet_manager.cc:1398] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240501 08:12:38.464627 19457 tablet_bootstrap.cc:492] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:38.469931 19445 tablet_bootstrap.cc:492] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:38.470208 19445 ts_tablet_manager.cc:1367] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.000s
I20240501 08:12:38.470614 19457 tablet_bootstrap.cc:654] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:38.471796 19413 tablet_bootstrap.cc:492] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:38.473425 19413 tablet_bootstrap.cc:654] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:38.473361 19445 raft_consensus.cc:348] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:38.473558 19445 raft_consensus.cc:374] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.473613 19445 raft_consensus.cc:729] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.473752 19447 tablet_bootstrap.cc:492] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:38.475227 19445 consensus_queue.cc:260] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:38.476598 19447 ts_tablet_manager.cc:1367] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.006s
I20240501 08:12:38.477399 19447 raft_consensus.cc:348] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.477789 19447 raft_consensus.cc:374] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.477916 19447 raft_consensus.cc:729] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.478236 19447 consensus_queue.cc:260] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.479259 19447 ts_tablet_manager.cc:1398] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:38.479728 19445 ts_tablet_manager.cc:1398] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:38.489904 19413 tablet_bootstrap.cc:492] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:38.490051 19447 tablet_bootstrap.cc:492] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:38.490221 19413 ts_tablet_manager.cc:1367] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240501 08:12:38.491199 19413 raft_consensus.cc:348] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.492173 19413 raft_consensus.cc:374] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.492347 19413 raft_consensus.cc:729] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.492763 19413 consensus_queue.cc:260] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.493690 19447 tablet_bootstrap.cc:654] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:38.496093 19413 ts_tablet_manager.cc:1398] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:38.497128 19444 tablet_bootstrap.cc:492] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:38.498943 19444 tablet_bootstrap.cc:654] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:38.503548 19457 tablet_bootstrap.cc:492] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:38.503829 19457 ts_tablet_manager.cc:1367] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.005s
I20240501 08:12:38.504626 19457 raft_consensus.cc:348] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.504927 19457 raft_consensus.cc:374] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.505035 19457 raft_consensus.cc:729] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.505314 19457 consensus_queue.cc:260] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.505771 19447 tablet_bootstrap.cc:492] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:38.506011 19447 ts_tablet_manager.cc:1367] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20240501 08:12:38.506822 19447 raft_consensus.cc:348] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.507099 19447 raft_consensus.cc:374] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.507200 19447 raft_consensus.cc:729] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.507905 19457 ts_tablet_manager.cc:1398] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:12:38.509936 19447 consensus_queue.cc:260] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.513793 19018 raft_consensus.cc:482] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:38.514019 19018 raft_consensus.cc:504] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.515050 19018 leader_election.cc:288] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:38.515353 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ef1222e2324309a009a657ca704795" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:38.515664 19447 ts_tablet_manager.cc:1398] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240501 08:12:38.516052 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ef1222e2324309a009a657ca704795" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:38.516263 15756 raft_consensus.cc:2454] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:38.516803 19444 tablet_bootstrap.cc:492] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:38.516842 15825 leader_election.cc:302] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:38.516997 19444 ts_tablet_manager.cc:1367] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.001s
I20240501 08:12:38.517204 19018 raft_consensus.cc:2781] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:38.517292 19018 raft_consensus.cc:482] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:38.517396 19018 raft_consensus.cc:3037] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.517835 19444 raft_consensus.cc:348] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:38.518111 19444 raft_consensus.cc:374] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.518280 19444 raft_consensus.cc:729] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.518553 19444 consensus_queue.cc:260] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:38.520100 19018 raft_consensus.cc:504] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
W20240501 08:12:38.522601 15826 leader_election.cc:341] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:38.524432 19444 ts_tablet_manager.cc:1398] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:12:38.524739 19444 tablet_bootstrap.cc:492] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:38.526087 19444 tablet_bootstrap.cc:654] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:38.527863 19018 leader_election.cc:288] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:38.528467 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ef1222e2324309a009a657ca704795" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:38.528676 15756 raft_consensus.cc:3037] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.534479 15756 raft_consensus.cc:2454] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:38.535207 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ef1222e2324309a009a657ca704795" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
W20240501 08:12:38.535807 15826 leader_election.cc:341] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:38.536470 15825 leader_election.cc:302] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters: 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:38.544500 19018 raft_consensus.cc:2781] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:38.544708 19018 raft_consensus.cc:686] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:38.544971 19018 consensus_queue.cc:237] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.560642 19444 tablet_bootstrap.cc:492] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:38.560804 19444 ts_tablet_manager.cc:1367] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.019s
I20240501 08:12:38.561575 19444 raft_consensus.cc:348] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.561808 19444 raft_consensus.cc:374] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.561861 19444 raft_consensus.cc:729] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.562062 19444 consensus_queue.cc:260] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.562697 19444 ts_tablet_manager.cc:1398] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:38.562949 19444 tablet_bootstrap.cc:492] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:38.565543 19444 tablet_bootstrap.cc:654] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:38.565992 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:38.584087 19444 tablet_bootstrap.cc:492] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:38.585076 19444 ts_tablet_manager.cc:1367] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.004s
I20240501 08:12:38.585800 19444 raft_consensus.cc:348] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.585991 19444 raft_consensus.cc:374] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:38.586051 19444 raft_consensus.cc:729] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:38.586266 19444 consensus_queue.cc:260] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.586783 19444 ts_tablet_manager.cc:1398] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:38.591356 19440 tablet.cc:1722] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:38.592020 19440 tablet.cc:1620] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:12:38.597514 19441 tablet.cc:1722] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:38.598273 19441 tablet.cc:1620] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:38.598742 19442 tablet.cc:1722] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:38.599378 19442 tablet.cc:1620] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:38.600598 19279 raft_consensus.cc:482] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:38.600829 19279 raft_consensus.cc:504] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:38.601617 19279 leader_election.cc:288] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:38.611663 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6339e48a2f4b9e94e32e05c7bf343b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:38.611987 13232 raft_consensus.cc:2454] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:38.614742 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6339e48a2f4b9e94e32e05c7bf343b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:38.615042 12971 raft_consensus.cc:2454] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:38.633390 19233 raft_consensus.cc:482] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:38.633625 19233 raft_consensus.cc:504] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.634761 19233 leader_election.cc:288] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:38.636004 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=78891ec772a2428091464473d9c4b9d5] alter complete (version 2)
I20240501 08:12:38.636041 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09d4bf8b82941199313f03bc1abeab5" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:38.637352 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09d4bf8b82941199313f03bc1abeab5" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:38.637653 15756 raft_consensus.cc:2454] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:38.638178 13300 leader_election.cc:302] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:38.638651 19233 raft_consensus.cc:2781] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:38.638748 19233 raft_consensus.cc:482] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:38.638828 19233 raft_consensus.cc:3037] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.638955 15960 leader_election.cc:302] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:38.640435 19404 raft_consensus.cc:2781] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:38.640591 19404 raft_consensus.cc:482] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:38.640688 19404 raft_consensus.cc:3037] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.642624 19233 raft_consensus.cc:504] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.646188 19233 leader_election.cc:288] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:38.660666 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09d4bf8b82941199313f03bc1abeab5" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:38.661000 15756 raft_consensus.cc:3037] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.665746 15756 raft_consensus.cc:2454] T d09d4bf8b82941199313f03bc1abeab5 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:38.636262 13232 raft_consensus.cc:2454] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:38.645475 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d09d4bf8b82941199313f03bc1abeab5" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:38.666594 13235 raft_consensus.cc:3037] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.670369 13235 raft_consensus.cc:2454] T d09d4bf8b82941199313f03bc1abeab5 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:38.671422 13301 leader_election.cc:302] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:38.683351 19393 raft_consensus.cc:2781] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:38.683554 19393 raft_consensus.cc:686] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:38.683917 19393 consensus_queue.cc:237] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.714576 12833 catalog_manager.cc:5484] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:38.779091 19404 raft_consensus.cc:504] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:38.787766 19404 leader_election.cc:288] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:38.788805 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6339e48a2f4b9e94e32e05c7bf343b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:38.789111 12971 raft_consensus.cc:3037] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.794931 12971 raft_consensus.cc:2454] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:38.796514 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca6339e48a2f4b9e94e32e05c7bf343b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:38.796799 13232 raft_consensus.cc:3037] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.812808 15960 leader_election.cc:302] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:38.814172 19279 raft_consensus.cc:2781] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:38.814473 19279 raft_consensus.cc:686] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:38.814873 19279 consensus_queue.cc:237] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:38.830760 13232 raft_consensus.cc:2454] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:38.853163 12832 catalog_manager.cc:5484] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:38.872207 12833 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:38.887840 19440 tablet.cc:1722] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:38.899886 12832 catalog_manager.cc:5484] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:38.903180 13363 consensus_queue.cc:237] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.916682 12831 catalog_manager.cc:5576] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301) does not have the latest schema for tablet 33ea19edb3b54fc48962f89532b4f36d (table rebalancer_dynamic_table_new_name [id=78891ec772a2428091464473d9c4b9d5]). Expected version 3 got 2
I20240501 08:12:38.920172 15756 raft_consensus.cc:1261] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.923074 19393 consensus_queue.cc:1035] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.931250 19455 raft_consensus.cc:482] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:38.931500 19455 raft_consensus.cc:504] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:12:38.932636 19455 leader_election.cc:288] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:38.932857 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ef1222e2324309a009a657ca704795" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:38.933782 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ef1222e2324309a009a657ca704795" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:38.934060 15755 raft_consensus.cc:2379] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ed448d04e4646658c0de37d829ff15b in current term 1: Already voted for candidate 4d516916eeac468db514eeb928f06190 in this term.
I20240501 08:12:38.936093 13170 leader_election.cc:302] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b; no voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:38.937652 19455 raft_consensus.cc:3037] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:38.944975 19455 raft_consensus.cc:2726] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240501 08:12:38.949616 13038 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 62489us
I20240501 08:12:38.949563 13232 raft_consensus.cc:1261] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:38.951550 19442 tablet.cc:1722] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:38.953547 19393 consensus_queue.cc:1035] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:38.954134 12832 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=78891ec772a2428091464473d9c4b9d5] alter complete (version 3)
I20240501 08:12:38.959715 19441 tablet.cc:1722] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:38.962162 19233 raft_consensus.cc:2932] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:38.970724 15756 raft_consensus.cc:2932] T faa5b3517ffc490cab68d09bbe2d5c0e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:38.973387 12830 catalog_manager.cc:5484] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:38.974429 13235 raft_consensus.cc:2932] T faa5b3517ffc490cab68d09bbe2d5c0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:12:38.984876 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet faa5b3517ffc490cab68d09bbe2d5c0e with delete_type TABLET_DATA_TOMBSTONED (TS e3fefd4dca2d41678187047f06c10187 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:38.987664 19487 tablet_replica.cc:330] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:38.987881 19487 raft_consensus.cc:2229] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:38.988137 19487 raft_consensus.cc:2258] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:38.988801 19487 ts_tablet_manager.cc:1875] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:38.995244 19487 ts_tablet_manager.cc:1888] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:38.995549 19487 log.cc:1196] T faa5b3517ffc490cab68d09bbe2d5c0e P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/faa5b3517ffc490cab68d09bbe2d5c0e
I20240501 08:12:38.996419 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet faa5b3517ffc490cab68d09bbe2d5c0e (table rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]) successfully deleted
I20240501 08:12:39.023291 19018 consensus_queue.cc:1035] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:39.043344 19436 consensus_queue.cc:1035] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:39.059840 12830 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:39.060086 12830 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:39.070576 12830 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 33ea19edb3b54fc48962f89532b4f36d
I20240501 08:12:39.071349 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ea19edb3b54fc48962f89532b4f36d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:39 UTC) from {username='slave'} at 127.0.0.1:58510
I20240501 08:12:39.072350 19487 tablet_replica.cc:330] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:12:39.072546 19487 raft_consensus.cc:2229] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:39.072855 19487 raft_consensus.cc:2258] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.073544 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ea19edb3b54fc48962f89532b4f36d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:39 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:39.075968 16137 tablet_service.cc:1514] Processing DeleteTablet for tablet 33ea19edb3b54fc48962f89532b4f36d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:39 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:39.079485 19487 ts_tablet_manager.cc:1875] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:39.100106 19393 consensus_queue.cc:1035] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:39.107887 19487 ts_tablet_manager.cc:1888] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:39.112550 19487 log.cc:1196] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/33ea19edb3b54fc48962f89532b4f36d
I20240501 08:12:39.113022 19487 ts_tablet_manager.cc:1909] T 33ea19edb3b54fc48962f89532b4f36d P e3fefd4dca2d41678187047f06c10187: Deleting consensus metadata
I20240501 08:12:39.114099 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 33ea19edb3b54fc48962f89532b4f36d (table rebalancer_dynamic_table_new_name [id=78891ec772a2428091464473d9c4b9d5]) successfully deleted
I20240501 08:12:39.114776 19503 tablet_replica.cc:330] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:39.115020 19503 raft_consensus.cc:2229] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.115262 19503 raft_consensus.cc:2258] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.116338 19503 ts_tablet_manager.cc:1875] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:39.116673 19502 tablet_replica.cc:330] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:39.116995 19502 raft_consensus.cc:2229] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:39.117251 19502 raft_consensus.cc:2258] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:39.118211 19502 ts_tablet_manager.cc:1875] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:39.124114 19503 ts_tablet_manager.cc:1888] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:39.124356 19503 log.cc:1196] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/33ea19edb3b54fc48962f89532b4f36d
I20240501 08:12:39.124429 19502 ts_tablet_manager.cc:1888] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:39.124569 19502 log.cc:1196] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/33ea19edb3b54fc48962f89532b4f36d
I20240501 08:12:39.125100 19502 ts_tablet_manager.cc:1909] T 33ea19edb3b54fc48962f89532b4f36d P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:39.126197 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 33ea19edb3b54fc48962f89532b4f36d (table rebalancer_dynamic_table_new_name [id=78891ec772a2428091464473d9c4b9d5]) successfully deleted
I20240501 08:12:39.128309 19503 ts_tablet_manager.cc:1909] T 33ea19edb3b54fc48962f89532b4f36d P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:39.129071 19393 consensus_queue.cc:1035] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:39.130146 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 33ea19edb3b54fc48962f89532b4f36d (table rebalancer_dynamic_table_new_name [id=78891ec772a2428091464473d9c4b9d5]) successfully deleted
I20240501 08:12:39.175004 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:39.187774 16137 tablet_service.cc:1467] Processing CreateTablet for tablet ea88204c356c47a7a9c283d60de3bbf4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f79e76287b14714a0ff05b4dc973164]), partition=
I20240501 08:12:39.188323 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea88204c356c47a7a9c283d60de3bbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.188881 16004 tablet_service.cc:1467] Processing CreateTablet for tablet ea88204c356c47a7a9c283d60de3bbf4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f79e76287b14714a0ff05b4dc973164]), partition=
I20240501 08:12:39.189240 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea88204c356c47a7a9c283d60de3bbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.193305 13344 tablet_service.cc:1467] Processing CreateTablet for tablet ea88204c356c47a7a9c283d60de3bbf4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7f79e76287b14714a0ff05b4dc973164]), partition=
I20240501 08:12:39.193696 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea88204c356c47a7a9c283d60de3bbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.206557 19538 tablet_bootstrap.cc:492] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:39.208176 19538 tablet_bootstrap.cc:654] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.279897 19543 tablet_bootstrap.cc:492] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:39.285400 19543 tablet_bootstrap.cc:654] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.299065 19543 tablet_bootstrap.cc:492] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:39.299911 19543 ts_tablet_manager.cc:1367] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.002s
I20240501 08:12:39.301131 19543 raft_consensus.cc:348] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.302631 19543 raft_consensus.cc:374] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.303340 19543 raft_consensus.cc:729] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.304143 19543 consensus_queue.cc:260] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.309336 19543 ts_tablet_manager.cc:1398] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240501 08:12:39.317665 19404 consensus_queue.cc:1035] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:39.322755 18749 consensus_queue.cc:1035] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:39.423713 19539 tablet_bootstrap.cc:492] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:39.425186 19539 tablet_bootstrap.cc:654] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
W20240501 08:12:39.435658 16090 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 75919.5us
I20240501 08:12:39.438589 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:39.451063 13344 tablet_service.cc:1467] Processing CreateTablet for tablet a713c94c6bb14d8793ee429e52f0e7a0 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:39.451572 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a713c94c6bb14d8793ee429e52f0e7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.451586 12953 tablet_service.cc:1467] Processing CreateTablet for tablet d93c796748404ca096aa600c94d5d5de (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:39.451980 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93c796748404ca096aa600c94d5d5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.455642 16004 tablet_service.cc:1467] Processing CreateTablet for tablet a713c94c6bb14d8793ee429e52f0e7a0 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:39.456137 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a713c94c6bb14d8793ee429e52f0e7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.459785 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 949d6765e68146d998ff4f5d2b120d68 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:39.460238 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949d6765e68146d998ff4f5d2b120d68. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 08:12:39.460700 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.69:42897, user_credentials={real_user=slave}} blocked reactor thread for 99956.7us
I20240501 08:12:39.464964 13084 tablet_service.cc:1467] Processing CreateTablet for tablet d93c796748404ca096aa600c94d5d5de (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:39.465389 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93c796748404ca096aa600c94d5d5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.468636 13085 tablet_service.cc:1467] Processing CreateTablet for tablet 949d6765e68146d998ff4f5d2b120d68 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:39.471700 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949d6765e68146d998ff4f5d2b120d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.471635 13343 tablet_service.cc:1467] Processing CreateTablet for tablet d93c796748404ca096aa600c94d5d5de (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:39.472132 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93c796748404ca096aa600c94d5d5de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.477118 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 949d6765e68146d998ff4f5d2b120d68 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:39.477595 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949d6765e68146d998ff4f5d2b120d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.487975 19538 tablet_bootstrap.cc:492] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:39.490971 16137 tablet_service.cc:1467] Processing CreateTablet for tablet a713c94c6bb14d8793ee429e52f0e7a0 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:39.492468 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a713c94c6bb14d8793ee429e52f0e7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:39.502122 19567 tablet_bootstrap.cc:492] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:39.488152 19538 ts_tablet_manager.cc:1367] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.282s user 0.068s sys 0.113s
I20240501 08:12:39.518779 19538 raft_consensus.cc:348] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.519014 19538 raft_consensus.cc:374] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.519078 19538 raft_consensus.cc:729] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.519613 19539 tablet_bootstrap.cc:492] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:39.520488 19567 tablet_bootstrap.cc:654] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.528528 19538 consensus_queue.cc:260] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.529439 19543 tablet_bootstrap.cc:492] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:39.531160 19543 tablet_bootstrap.cc:654] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.536528 19539 ts_tablet_manager.cc:1367] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.000s
I20240501 08:12:39.538077 19567 tablet_bootstrap.cc:492] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:39.538218 19567 ts_tablet_manager.cc:1367] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240501 08:12:39.539381 19567 raft_consensus.cc:348] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.539568 19567 raft_consensus.cc:374] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.539615 19567 raft_consensus.cc:729] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.540774 19567 consensus_queue.cc:260] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.542780 19567 ts_tablet_manager.cc:1398] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:39.543049 19567 tablet_bootstrap.cc:492] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:39.544440 19538 ts_tablet_manager.cc:1398] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20240501 08:12:39.546223 19538 tablet_bootstrap.cc:492] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:39.548388 19538 tablet_bootstrap.cc:654] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.548470 19567 tablet_bootstrap.cc:654] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.552261 19568 tablet_bootstrap.cc:492] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:39.554379 19568 tablet_bootstrap.cc:654] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.554903 19563 tablet_bootstrap.cc:492] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:39.558225 19539 raft_consensus.cc:348] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.559262 19539 raft_consensus.cc:374] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.559762 19539 raft_consensus.cc:729] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.562423 19539 consensus_queue.cc:260] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.563089 19538 tablet_bootstrap.cc:492] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:39.563225 19538 ts_tablet_manager.cc:1367] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240501 08:12:39.563921 19538 raft_consensus.cc:348] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.564106 19538 raft_consensus.cc:374] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.564153 19538 raft_consensus.cc:729] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.564671 19566 tablet_bootstrap.cc:492] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:39.566207 19538 consensus_queue.cc:260] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.567981 19539 ts_tablet_manager.cc:1398] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240501 08:12:39.568147 19538 ts_tablet_manager.cc:1398] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:39.568545 19538 tablet_bootstrap.cc:492] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:39.570415 19566 tablet_bootstrap.cc:654] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.574697 19539 tablet_bootstrap.cc:492] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:39.575505 19543 tablet_bootstrap.cc:492] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:39.575738 19543 ts_tablet_manager.cc:1367] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.006s
I20240501 08:12:39.575907 19563 tablet_bootstrap.cc:654] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.576690 19543 raft_consensus.cc:348] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.576978 19543 raft_consensus.cc:374] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.577077 19543 raft_consensus.cc:729] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.577368 19543 consensus_queue.cc:260] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.578262 19538 tablet_bootstrap.cc:654] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.580192 19543 ts_tablet_manager.cc:1398] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:39.583416 19566 tablet_bootstrap.cc:492] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:39.583583 19566 ts_tablet_manager.cc:1367] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240501 08:12:39.584436 19566 raft_consensus.cc:348] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.584643 19566 raft_consensus.cc:374] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.584739 19566 raft_consensus.cc:729] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.585021 19566 consensus_queue.cc:260] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.591048 19539 tablet_bootstrap.cc:654] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:39.591223 19566 ts_tablet_manager.cc:1398] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 08:12:39.615584 19567 tablet_bootstrap.cc:492] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:39.615738 19567 ts_tablet_manager.cc:1367] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.073s user 0.001s sys 0.031s
I20240501 08:12:39.615721 18749 raft_consensus.cc:482] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:39.615911 18749 raft_consensus.cc:504] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.616489 19567 raft_consensus.cc:348] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.616612 18749 leader_election.cc:288] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:39.616660 19567 raft_consensus.cc:374] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.616701 19567 raft_consensus.cc:729] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.616900 19567 consensus_queue.cc:260] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.617947 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea88204c356c47a7a9c283d60de3bbf4" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:39.618180 16154 raft_consensus.cc:2454] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:39.618569 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea88204c356c47a7a9c283d60de3bbf4" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:39.618809 13364 raft_consensus.cc:2454] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:39.619658 15959 leader_election.cc:302] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:39.622076 18749 raft_consensus.cc:2781] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:39.622203 18749 raft_consensus.cc:482] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:39.622313 18749 raft_consensus.cc:3037] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.624094 19480 raft_consensus.cc:482] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:39.624228 19480 raft_consensus.cc:504] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.624845 19480 leader_election.cc:288] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:39.625501 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:39.625723 15892 raft_consensus.cc:2454] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:39.626407 18749 raft_consensus.cc:504] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.626906 13038 leader_election.cc:302] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:39.627367 19567 ts_tablet_manager.cc:1398] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240501 08:12:39.627350 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:39.627578 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea88204c356c47a7a9c283d60de3bbf4" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:39.627769 16154 raft_consensus.cc:3037] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
W20240501 08:12:39.627948 13040 leader_election.cc:341] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:39.628408 19480 raft_consensus.cc:2781] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:39.628433 19538 tablet_bootstrap.cc:492] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:39.628486 19480 raft_consensus.cc:482] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:39.628551 19480 raft_consensus.cc:3037] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.628583 19538 ts_tablet_manager.cc:1367] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.005s
I20240501 08:12:39.629650 19538 raft_consensus.cc:348] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.629820 19538 raft_consensus.cc:374] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.629889 19538 raft_consensus.cc:729] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.630089 19538 consensus_queue.cc:260] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.630493 16154 raft_consensus.cc:2454] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:39.630690 19538 ts_tablet_manager.cc:1398] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:39.630714 19480 raft_consensus.cc:504] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.630942 15957 leader_election.cc:302] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:39.631673 19477 raft_consensus.cc:2781] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:39.631819 19477 raft_consensus.cc:686] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:39.632453 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea88204c356c47a7a9c283d60de3bbf4" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:39.632635 13364 raft_consensus.cc:3037] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.632126 19477 consensus_queue.cc:237] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.635078 19233 raft_consensus.cc:482] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:39.635200 19233 raft_consensus.cc:504] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.635942 19233 leader_election.cc:288] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:39.636567 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c796748404ca096aa600c94d5d5de" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
W20240501 08:12:39.637130 13301 leader_election.cc:341] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:39.637441 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c796748404ca096aa600c94d5d5de" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:39.637631 13104 raft_consensus.cc:2454] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:39.639545 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:39.639825 15892 raft_consensus.cc:3037] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.642812 15892 raft_consensus.cc:2454] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:39.643244 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
W20240501 08:12:39.643932 13040 leader_election.cc:341] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:39.644003 13364 raft_consensus.cc:2454] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:39.644320 18749 leader_election.cc:288] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:39.644215 13300 leader_election.cc:302] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:39.644824 19233 raft_consensus.cc:2781] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:39.644912 19233 raft_consensus.cc:482] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:39.644992 19233 raft_consensus.cc:3037] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.646430 19539 tablet_bootstrap.cc:492] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:39.646605 19539 ts_tablet_manager.cc:1367] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20240501 08:12:39.647509 19539 raft_consensus.cc:348] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.647794 19539 raft_consensus.cc:374] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.647888 19539 raft_consensus.cc:729] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.648131 19539 consensus_queue.cc:260] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.648716 19539 ts_tablet_manager.cc:1398] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:39.648870 19233 raft_consensus.cc:504] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.649845 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c796748404ca096aa600c94d5d5de" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
W20240501 08:12:39.650485 13301 leader_election.cc:341] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:39.652685 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c796748404ca096aa600c94d5d5de" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:39.652952 13104 raft_consensus.cc:3037] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.656404 19233 leader_election.cc:288] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:39.657302 19568 tablet_bootstrap.cc:492] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:39.657536 19568 ts_tablet_manager.cc:1367] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.105s user 0.000s sys 0.010s
I20240501 08:12:39.661041 19568 raft_consensus.cc:348] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.661401 19568 raft_consensus.cc:374] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.661553 19568 raft_consensus.cc:729] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.662021 19568 consensus_queue.cc:260] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.665627 19568 ts_tablet_manager.cc:1398] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 08:12:39.666615 19563 tablet_bootstrap.cc:492] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:39.666843 19563 ts_tablet_manager.cc:1367] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.112s user 0.005s sys 0.006s
I20240501 08:12:39.669950 19563 raft_consensus.cc:348] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.670861 19563 raft_consensus.cc:374] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:39.670989 19563 raft_consensus.cc:729] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:39.672159 19480 leader_election.cc:288] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:39.672638 13104 raft_consensus.cc:2454] T d93c796748404ca096aa600c94d5d5de P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:39.672928 13038 leader_election.cc:302] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters: 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:39.673465 19480 raft_consensus.cc:2781] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:39.673643 19480 raft_consensus.cc:686] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:39.673969 19480 consensus_queue.cc:237] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.675894 13300 leader_election.cc:302] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:39.676527 19233 raft_consensus.cc:2781] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:39.676784 19233 raft_consensus.cc:686] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:39.677059 19233 consensus_queue.cc:237] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.678556 19563 consensus_queue.cc:260] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.679446 19563 ts_tablet_manager.cc:1398] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240501 08:12:39.704942 12826 catalog_manager.cc:5484] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:39.710088 19277 raft_consensus.cc:482] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:39.710317 19277 raft_consensus.cc:504] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:39.711304 19277 leader_election.cc:288] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:39.711932 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:39.712363 15892 raft_consensus.cc:2379] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 982f49fc87f149439cda0de2261b6bfd in current term 1: Already voted for candidate e3fefd4dca2d41678187047f06c10187 in this term.
I20240501 08:12:39.714227 12826 catalog_manager.cc:5484] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:39.714370 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:39.715808 15691 leader_election.cc:302] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187
I20240501 08:12:39.716401 19277 raft_consensus.cc:3037] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.721175 19277 raft_consensus.cc:2726] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:39.721653 12826 catalog_manager.cc:5484] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:39.727644 15870 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:39.732923 19233 raft_consensus.cc:482] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:39.733103 19233 raft_consensus.cc:504] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.735736 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:39.736064 16154 raft_consensus.cc:2454] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
I20240501 08:12:39.736575 13298 leader_election.cc:302] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:39.736943 19393 raft_consensus.cc:2781] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:39.737025 19393 raft_consensus.cc:482] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:39.737093 19393 raft_consensus.cc:3037] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.740947 19393 raft_consensus.cc:504] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.742237 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:39.742527 16154 raft_consensus.cc:3037] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.745833 16154 raft_consensus.cc:2454] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:39.746330 13298 leader_election.cc:302] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:39.756994 19393 leader_election.cc:288] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:39.757493 19393 raft_consensus.cc:2781] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:39.757612 19393 raft_consensus.cc:686] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:39.757964 19393 consensus_queue.cc:237] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:39.790620 19233 leader_election.cc:288] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
W20240501 08:12:39.817780 13298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 71478us
I20240501 08:12:39.820916 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:12:39.820659 12826 catalog_manager.cc:5484] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:39.858803 16004 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240501 08:12:39.866530 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:39.866781 16024 raft_consensus.cc:3037] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:39.869762 16024 raft_consensus.cc:2454] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:39.870149 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:39.870378 16022 raft_consensus.cc:2362] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1. Re-sending same reply.
I20240501 08:12:39.873394 13213 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:39.873836 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:39.884915 16157 raft_consensus.cc:1261] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:39.885576 19477 consensus_queue.cc:1035] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:39.905938 13364 raft_consensus.cc:1261] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:39.907181 19477 consensus_queue.cc:1035] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:39.912925 13298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 59852.1us
I20240501 08:12:39.971570 12833 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet ea88204c356c47a7a9c283d60de3bbf4 (table rebalancer_dynamic_table [id=7f79e76287b14714a0ff05b4dc973164]). Expected version 1 got 0
I20240501 08:12:39.973045 19610 tablet.cc:1722] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:39.974150 19610 tablet.cc:1620] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:39.980417 12833 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet ea88204c356c47a7a9c283d60de3bbf4 (table rebalancer_dynamic_table [id=7f79e76287b14714a0ff05b4dc973164]). Expected version 1 got 0
I20240501 08:12:39.990763 16137 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:39.994138 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=7f79e76287b14714a0ff05b4dc973164] alter complete (version 1)
I20240501 08:12:40.033880 12833 catalog_manager.cc:5576] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383) does not have the latest schema for tablet ea88204c356c47a7a9c283d60de3bbf4 (table rebalancer_dynamic_table [id=7f79e76287b14714a0ff05b4dc973164]). Expected version 1 got 0
I20240501 08:12:40.044909 19611 tablet.cc:1722] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:40.051368 19612 tablet.cc:1722] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:40.052942 19612 tablet.cc:1620] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:40.055400 19611 tablet.cc:1620] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:40.071084 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:40.073446 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:40.083443 13084 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:12:40.091501 19591 consensus_queue.cc:1035] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:40.103884 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:40.120452 19610 tablet.cc:1722] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:40.121031 19610 tablet.cc:1620] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:40.125993 19612 tablet.cc:1722] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:40.129588 19612 tablet.cc:1620] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: MemRowSet was empty: no flush needed.
I20240501 08:12:40.130398 12833 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet ea88204c356c47a7a9c283d60de3bbf4 (table rebalancer_dynamic_table [id=7f79e76287b14714a0ff05b4dc973164]). Expected version 2 got 1
I20240501 08:12:40.137467 19148 raft_consensus.cc:482] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:40.137652 19148 raft_consensus.cc:504] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:40.139669 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=7f79e76287b14714a0ff05b4dc973164] alter complete (version 2)
I20240501 08:12:40.139990 19148 leader_election.cc:288] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:40.155328 19611 tablet.cc:1722] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:40.155939 19611 tablet.cc:1620] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:40.172603 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c796748404ca096aa600c94d5d5de" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:40.173779 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93c796748404ca096aa600c94d5d5de" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:40.178974 12909 leader_election.cc:302] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2; no voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187
I20240501 08:12:40.182896 19148 raft_consensus.cc:2726] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:40.190466 12974 raft_consensus.cc:3037] T d93c796748404ca096aa600c94d5d5de P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.199225 19591 consensus_queue.cc:1035] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:40.210070 19591 consensus_queue.cc:1035] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:40.224030 19393 consensus_queue.cc:1035] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:40.226171 19480 consensus_queue.cc:1035] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:40.242985 12833 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:40.255674 19610 tablet.cc:1722] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:40.262092 19611 tablet.cc:1722] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:40.272594 12833 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=7f79e76287b14714a0ff05b4dc973164] alter complete (version 3)
I20240501 08:12:40.282912 19612 tablet.cc:1722] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:40.312019 19624 consensus_queue.cc:1035] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:40.383172 12833 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:40.383472 12833 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:40.387343 12833 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet ea88204c356c47a7a9c283d60de3bbf4
I20240501 08:12:40.388742 16004 tablet_service.cc:1514] Processing DeleteTablet for tablet ea88204c356c47a7a9c283d60de3bbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:40 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:40.388839 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet ea88204c356c47a7a9c283d60de3bbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:40 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:40.388924 16137 tablet_service.cc:1514] Processing DeleteTablet for tablet ea88204c356c47a7a9c283d60de3bbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:40 UTC) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:40.393682 19644 tablet_replica.cc:330] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:40.393949 19644 raft_consensus.cc:2229] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.394192 19644 raft_consensus.cc:2258] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.396087 19644 ts_tablet_manager.cc:1875] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:40.407943 19645 tablet_replica.cc:330] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:40.408531 19645 raft_consensus.cc:2229] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.409292 19645 raft_consensus.cc:2258] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.409451 19643 tablet_replica.cc:330] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:40.409682 19643 raft_consensus.cc:2229] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:40.410077 19643 raft_consensus.cc:2258] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.410928 19645 ts_tablet_manager.cc:1875] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:40.411002 19643 ts_tablet_manager.cc:1875] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:40.411088 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:40.413705 19644 ts_tablet_manager.cc:1888] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:40.413899 19644 log.cc:1196] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/ea88204c356c47a7a9c283d60de3bbf4
I20240501 08:12:40.414373 19644 ts_tablet_manager.cc:1909] T ea88204c356c47a7a9c283d60de3bbf4 P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:40.416896 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet ea88204c356c47a7a9c283d60de3bbf4 (table rebalancer_dynamic_table_new_name [id=7f79e76287b14714a0ff05b4dc973164]) successfully deleted
I20240501 08:12:40.423241 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 4345eabfdb894def8d7063773dc5fac6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:40.423586 16137 tablet_service.cc:1467] Processing CreateTablet for tablet 4345eabfdb894def8d7063773dc5fac6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:40.423725 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4345eabfdb894def8d7063773dc5fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.424000 16137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4345eabfdb894def8d7063773dc5fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.425801 15736 tablet_service.cc:1467] Processing CreateTablet for tablet 98306f9c17d8420396452ab3854d5246 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:40.426239 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98306f9c17d8420396452ab3854d5246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.426565 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 0f723c6321424530bcd41b9ecdcbe0b3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:40.427008 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f723c6321424530bcd41b9ecdcbe0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.426978 16004 tablet_service.cc:1467] Processing CreateTablet for tablet 4345eabfdb894def8d7063773dc5fac6 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:40.427290 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 98306f9c17d8420396452ab3854d5246 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:40.427423 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4345eabfdb894def8d7063773dc5fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.427927 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98306f9c17d8420396452ab3854d5246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.437189 13084 tablet_service.cc:1467] Processing CreateTablet for tablet 0f723c6321424530bcd41b9ecdcbe0b3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:40.437778 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f723c6321424530bcd41b9ecdcbe0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.439683 16138 tablet_service.cc:1467] Processing CreateTablet for tablet 98306f9c17d8420396452ab3854d5246 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:40.440183 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98306f9c17d8420396452ab3854d5246. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.441357 19643 ts_tablet_manager.cc:1888] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:40.441501 19643 log.cc:1196] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/ea88204c356c47a7a9c283d60de3bbf4
I20240501 08:12:40.441968 19643 ts_tablet_manager.cc:1909] T ea88204c356c47a7a9c283d60de3bbf4 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:40.443356 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 0f723c6321424530bcd41b9ecdcbe0b3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:40.443495 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet ea88204c356c47a7a9c283d60de3bbf4 (table rebalancer_dynamic_table_new_name [id=7f79e76287b14714a0ff05b4dc973164]) successfully deleted
I20240501 08:12:40.443868 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f723c6321424530bcd41b9ecdcbe0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.462386 19647 tablet_bootstrap.cc:492] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:40.467389 19651 tablet_bootstrap.cc:492] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:40.469368 19651 tablet_bootstrap.cc:654] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.470669 19647 tablet_bootstrap.cc:654] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.480232 19647 tablet_bootstrap.cc:492] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:40.480875 19651 tablet_bootstrap.cc:492] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:40.481166 19647 ts_tablet_manager.cc:1367] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 08:12:40.481520 19651 ts_tablet_manager.cc:1367] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240501 08:12:40.482547 19647 raft_consensus.cc:348] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.483454 19652 tablet_bootstrap.cc:492] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:40.483821 19651 raft_consensus.cc:348] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.485164 19651 raft_consensus.cc:374] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.485357 19647 raft_consensus.cc:374] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.485632 19651 raft_consensus.cc:729] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.485801 19647 raft_consensus.cc:729] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.487668 19656 tablet_bootstrap.cc:492] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:40.488236 19653 tablet_bootstrap.cc:492] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:40.488821 19651 consensus_queue.cc:260] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.489320 19647 consensus_queue.cc:260] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.490042 19648 tablet_bootstrap.cc:492] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:40.493785 19647 ts_tablet_manager.cc:1398] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 08:12:40.494432 19651 ts_tablet_manager.cc:1398] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240501 08:12:40.494776 19647 tablet_bootstrap.cc:492] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:40.496533 19647 tablet_bootstrap.cc:654] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.502374 12833 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:40.526366 16024 consensus_queue.cc:237] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.528951 16154 raft_consensus.cc:1261] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.529481 19404 consensus_queue.cc:1035] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.532883 19404 raft_consensus.cc:2932] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:40.535181 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=389d409bbeec43a28e75d26b8d520887]), partition=
I20240501 08:12:40.535549 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.535687 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=389d409bbeec43a28e75d26b8d520887]), partition=
I20240501 08:12:40.536027 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.536449 15753 raft_consensus.cc:1261] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:40.538339 19477 consensus_queue.cc:1035] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:40.542116 16004 tablet_service.cc:1467] Processing CreateTablet for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=389d409bbeec43a28e75d26b8d520887]), partition=
I20240501 08:12:40.541734 16154 raft_consensus.cc:2932] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:40.542479 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:40.558301 19665 tablet_bootstrap.cc:492] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:40.560360 19665 tablet_bootstrap.cc:654] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.561869 12830 catalog_manager.cc:5484] T 82e08735b5db41dcbccec46aa7278530 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:40.569588 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e08735b5db41dcbccec46aa7278530 with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:40.576046 19645 ts_tablet_manager.cc:1888] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:40.579922 19656 tablet_bootstrap.cc:654] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.584014 19653 tablet_bootstrap.cc:654] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.594089 19652 tablet_bootstrap.cc:654] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.595355 19648 tablet_bootstrap.cc:654] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.576340 19645 log.cc:1196] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/ea88204c356c47a7a9c283d60de3bbf4
I20240501 08:12:40.601996 19645 ts_tablet_manager.cc:1909] T ea88204c356c47a7a9c283d60de3bbf4 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting consensus metadata
I20240501 08:12:40.603590 15753 raft_consensus.cc:2932] T 82e08735b5db41dcbccec46aa7278530 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:40.605305 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet ea88204c356c47a7a9c283d60de3bbf4 (table rebalancer_dynamic_table_new_name [id=7f79e76287b14714a0ff05b4dc973164]) successfully deleted
I20240501 08:12:40.606346 19656 tablet_bootstrap.cc:492] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:40.607355 19652 tablet_bootstrap.cc:492] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:40.607604 19652 ts_tablet_manager.cc:1367] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.124s user 0.004s sys 0.000s
I20240501 08:12:40.609547 19648 tablet_bootstrap.cc:492] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:40.609691 19648 ts_tablet_manager.cc:1367] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.121s user 0.003s sys 0.000s
I20240501 08:12:40.610606 19648 raft_consensus.cc:348] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.610836 19648 raft_consensus.cc:374] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.611603 19653 tablet_bootstrap.cc:492] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:40.611860 19656 ts_tablet_manager.cc:1367] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.124s user 0.001s sys 0.002s
I20240501 08:12:40.613641 19652 raft_consensus.cc:348] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.613817 19652 raft_consensus.cc:374] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.616514 19653 ts_tablet_manager.cc:1367] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.127s user 0.002s sys 0.002s
I20240501 08:12:40.618615 19653 raft_consensus.cc:348] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.616734 19648 raft_consensus.cc:729] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.621138 19665 tablet_bootstrap.cc:492] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:40.617228 19656 raft_consensus.cc:348] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.621270 19665 ts_tablet_manager.cc:1367] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.020s
I20240501 08:12:40.621479 19648 consensus_queue.cc:260] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.622114 19648 ts_tablet_manager.cc:1398] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 08:12:40.622713 19665 raft_consensus.cc:348] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.623427 19665 raft_consensus.cc:374] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.623600 19665 raft_consensus.cc:729] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.623898 19665 consensus_queue.cc:260] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.623958 19652 raft_consensus.cc:729] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.621258 19656 raft_consensus.cc:374] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.624439 19656 raft_consensus.cc:729] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.624912 19652 consensus_queue.cc:260] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.625351 19656 consensus_queue.cc:260] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.626277 19665 ts_tablet_manager.cc:1398] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:12:40.632618 19653 raft_consensus.cc:374] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.632846 19648 tablet_bootstrap.cc:492] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:40.634559 19648 tablet_bootstrap.cc:654] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.635716 19652 ts_tablet_manager.cc:1398] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20240501 08:12:40.635987 19652 tablet_bootstrap.cc:492] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:40.636422 19653 raft_consensus.cc:729] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.637102 19656 ts_tablet_manager.cc:1398] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240501 08:12:40.637785 19653 consensus_queue.cc:260] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.639205 19667 tablet_replica.cc:330] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:40.639322 19648 tablet_bootstrap.cc:492] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:40.639535 19648 ts_tablet_manager.cc:1367] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:12:40.639703 19667 raft_consensus.cc:2229] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:40.639958 19667 raft_consensus.cc:2258] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:40.640717 19648 raft_consensus.cc:348] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.640947 19653 ts_tablet_manager.cc:1398] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240501 08:12:40.641001 19648 raft_consensus.cc:374] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.641160 19648 raft_consensus.cc:729] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.641443 19648 consensus_queue.cc:260] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.642017 19648 ts_tablet_manager.cc:1398] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:40.643282 19667 ts_tablet_manager.cc:1875] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:40.654340 19647 tablet_bootstrap.cc:492] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:40.654580 19647 ts_tablet_manager.cc:1367] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.160s user 0.011s sys 0.036s
I20240501 08:12:40.655184 19623 raft_consensus.cc:482] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:40.655335 19623 raft_consensus.cc:504] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.655350 19647 raft_consensus.cc:348] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.655566 19647 raft_consensus.cc:374] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.655642 19647 raft_consensus.cc:729] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.655917 19647 consensus_queue.cc:260] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.656046 19623 leader_election.cc:288] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:40.656450 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f723c6321424530bcd41b9ecdcbe0b3" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:40.656670 19647 ts_tablet_manager.cc:1398] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:40.656724 15891 raft_consensus.cc:2454] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:40.657290 13038 leader_election.cc:302] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:40.657768 19647 tablet_bootstrap.cc:492] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:40.657805 19480 raft_consensus.cc:2781] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:40.657878 19480 raft_consensus.cc:482] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:40.657934 19480 raft_consensus.cc:3037] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.659377 19647 tablet_bootstrap.cc:654] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.660207 19652 tablet_bootstrap.cc:654] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.662570 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f723c6321424530bcd41b9ecdcbe0b3" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
W20240501 08:12:40.663154 13040 leader_election.cc:341] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:40.664878 19480 raft_consensus.cc:504] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.665891 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f723c6321424530bcd41b9ecdcbe0b3" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:40.666016 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f723c6321424530bcd41b9ecdcbe0b3" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:40.666136 15891 raft_consensus.cc:3037] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.666479 19647 tablet_bootstrap.cc:492] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:40.666620 19647 ts_tablet_manager.cc:1367] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
W20240501 08:12:40.666584 13040 leader_election.cc:341] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:40.668148 19667 ts_tablet_manager.cc:1888] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:40.668440 19667 log.cc:1196] T 82e08735b5db41dcbccec46aa7278530 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/82e08735b5db41dcbccec46aa7278530
I20240501 08:12:40.665478 19480 leader_election.cc:288] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:40.668932 15891 raft_consensus.cc:2454] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:40.669431 13038 leader_election.cc:302] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters: 5dd722c770d34d6c9154a87917117161
I20240501 08:12:40.669451 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 82e08735b5db41dcbccec46aa7278530 (table rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]) successfully deleted
I20240501 08:12:40.672394 19480 raft_consensus.cc:2781] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:40.672542 19480 raft_consensus.cc:686] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:40.672791 19480 consensus_queue.cc:237] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.674790 19647 raft_consensus.cc:348] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.675011 19647 raft_consensus.cc:374] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.675091 19647 raft_consensus.cc:729] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.675299 19647 consensus_queue.cc:260] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.676565 19652 tablet_bootstrap.cc:492] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:40.676744 19652 ts_tablet_manager.cc:1367] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.002s
I20240501 08:12:40.677526 19652 raft_consensus.cc:348] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.677743 19652 raft_consensus.cc:374] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.677815 19652 raft_consensus.cc:729] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.678090 19652 consensus_queue.cc:260] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.678720 19652 ts_tablet_manager.cc:1398] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:12:40.685043 19647 ts_tablet_manager.cc:1398] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240501 08:12:40.690800 19647 tablet_bootstrap.cc:492] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:40.692010 12833 catalog_manager.cc:5484] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:40.694531 19647 tablet_bootstrap.cc:654] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:40.716235 19647 tablet_bootstrap.cc:492] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:40.719310 19647 ts_tablet_manager.cc:1367] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.006s
I20240501 08:12:40.724606 19647 raft_consensus.cc:348] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.724948 19647 raft_consensus.cc:374] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:40.725039 19647 raft_consensus.cc:729] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:40.725322 19647 consensus_queue.cc:260] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.728880 19647 ts_tablet_manager.cc:1398] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:40.756783 19663 raft_consensus.cc:482] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:40.756996 19663 raft_consensus.cc:504] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.757714 19663 leader_election.cc:288] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:40.758163 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98306f9c17d8420396452ab3854d5246" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:40.758522 15753 raft_consensus.cc:2454] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:40.758577 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98306f9c17d8420396452ab3854d5246" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:40.758845 16157 raft_consensus.cc:2454] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:40.759070 15959 leader_election.cc:302] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:40.759362 19663 raft_consensus.cc:2781] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:40.759474 19663 raft_consensus.cc:482] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:40.759568 19663 raft_consensus.cc:3037] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.762413 19663 raft_consensus.cc:504] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.763072 19663 leader_election.cc:288] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:40.763639 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98306f9c17d8420396452ab3854d5246" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:40.763872 16157 raft_consensus.cc:3037] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.763904 15753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98306f9c17d8420396452ab3854d5246" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:40.764065 15753 raft_consensus.cc:3037] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.766537 15753 raft_consensus.cc:2454] T 98306f9c17d8420396452ab3854d5246 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:40.766834 16157 raft_consensus.cc:2454] T 98306f9c17d8420396452ab3854d5246 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:40.766891 15959 leader_election.cc:302] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:12:40.767210 19663 raft_consensus.cc:2781] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:40.767390 19663 raft_consensus.cc:686] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:40.767732 19663 consensus_queue.cc:237] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:40.771762 12830 catalog_manager.cc:5484] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:40.827852 19663 raft_consensus.cc:482] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:40.828086 19663 raft_consensus.cc:504] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.828899 19663 leader_election.cc:288] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:40.829710 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f723c6321424530bcd41b9ecdcbe0b3" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:40.829761 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f723c6321424530bcd41b9ecdcbe0b3" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:40.830006 15891 raft_consensus.cc:2379] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5dd722c770d34d6c9154a87917117161 in current term 1: Already voted for candidate e3fefd4dca2d41678187047f06c10187 in this term.
I20240501 08:12:40.830629 15957 leader_election.cc:302] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187
I20240501 08:12:40.832811 19663 raft_consensus.cc:3037] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.835718 19663 raft_consensus.cc:2726] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:40.898499 19663 raft_consensus.cc:482] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:40.898712 19663 raft_consensus.cc:504] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.899909 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4345eabfdb894def8d7063773dc5fac6" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:40.900008 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4345eabfdb894def8d7063773dc5fac6" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:40.900171 16154 raft_consensus.cc:2454] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:40.900225 12974 raft_consensus.cc:2454] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:40.900647 15957 leader_election.cc:302] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:40.900957 19404 raft_consensus.cc:2781] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:40.901077 19404 raft_consensus.cc:482] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:40.901165 19404 raft_consensus.cc:3037] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.913951 19663 leader_election.cc:288] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:40.904101 19404 raft_consensus.cc:504] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.915202 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4345eabfdb894def8d7063773dc5fac6" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:40.915333 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4345eabfdb894def8d7063773dc5fac6" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:40.915419 12974 raft_consensus.cc:3037] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.915555 16154 raft_consensus.cc:3037] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:40.918113 12974 raft_consensus.cc:2454] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:40.918599 15960 leader_election.cc:302] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:40.921124 16154 raft_consensus.cc:2454] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:40.931197 19663 raft_consensus.cc:2781] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:40.931548 19663 raft_consensus.cc:686] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:40.934201 19404 leader_election.cc:288] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:40.931849 19663 consensus_queue.cc:237] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:40.947016 12826 catalog_manager.cc:5484] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:41.085016 19730 raft_consensus.cc:482] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:41.085253 19646 raft_consensus.cc:482] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:41.085239 19730 raft_consensus.cc:504] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:41.085420 19646 raft_consensus.cc:504] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:41.086025 19730 leader_election.cc:288] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:41.086109 19646 leader_election.cc:288] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:41.086408 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fda9fa3ddf411c8efb1d82cfa7c0f0" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:41.086612 12974 raft_consensus.cc:2454] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:41.086616 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fda9fa3ddf411c8efb1d82cfa7c0f0" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:41.086745 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fda9fa3ddf411c8efb1d82cfa7c0f0" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:41.086799 16022 raft_consensus.cc:2454] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:41.086958 13235 raft_consensus.cc:2454] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:41.087019 13171 leader_election.cc:302] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:41.087275 19730 raft_consensus.cc:2781] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:41.087332 12910 leader_election.cc:302] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2; no voters:
I20240501 08:12:41.087388 19730 raft_consensus.cc:482] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:41.087471 19730 raft_consensus.cc:3037] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:41.087450 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fda9fa3ddf411c8efb1d82cfa7c0f0" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:41.087641 16022 raft_consensus.cc:2454] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 0.
I20240501 08:12:41.088125 19148 raft_consensus.cc:2781] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:41.088210 19148 raft_consensus.cc:482] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:41.088284 19148 raft_consensus.cc:3037] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:41.089949 19730 raft_consensus.cc:504] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:41.090240 19148 raft_consensus.cc:504] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:41.090620 19730 leader_election.cc:288] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:41.090814 19148 leader_election.cc:288] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Requested vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:41.090991 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fda9fa3ddf411c8efb1d82cfa7c0f0" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:41.091204 12974 raft_consensus.cc:2379] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ed448d04e4646658c0de37d829ff15b in current term 1: Already voted for candidate 76d830e199eb461c8ca689c9460ee0c2 in this term.
I20240501 08:12:41.092402 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fda9fa3ddf411c8efb1d82cfa7c0f0" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:41.092705 13235 raft_consensus.cc:2379] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 76d830e199eb461c8ca689c9460ee0c2 in current term 1: Already voted for candidate 6ed448d04e4646658c0de37d829ff15b in this term.
I20240501 08:12:41.092839 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fda9fa3ddf411c8efb1d82cfa7c0f0" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:41.093032 16022 raft_consensus.cc:3037] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:41.093384 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80fda9fa3ddf411c8efb1d82cfa7c0f0" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:41.093991 12909 leader_election.cc:302] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2; no voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:41.095599 16022 raft_consensus.cc:2454] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:41.095901 19646 raft_consensus.cc:2726] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 08:12:41.096006 13170 leader_election.cc:302] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:41.098176 19631 raft_consensus.cc:2781] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:41.098363 19631 raft_consensus.cc:686] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:41.098803 19631 consensus_queue.cc:237] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:41.104867 12830 catalog_manager.cc:5484] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:41.116187 19480 consensus_queue.cc:1035] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:41.156962 19624 consensus_queue.cc:1035] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:41.178369 15870 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240501 08:12:41.208986 16004 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:12:41.209826 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:12:41.224617 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:41.234160 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:41.237110 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:41.267401 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:41.277683 19404 consensus_queue.cc:1035] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:41.334692 19663 consensus_queue.cc:1035] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:41.365291 19404 consensus_queue.cc:1035] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:12:41.367136 15691 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 100884us
I20240501 08:12:41.367636 13084 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 08:12:41.379581 16022 raft_consensus.cc:1261] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:41.380154 19731 consensus_queue.cc:1035] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:41.380340 19477 consensus_queue.cc:1035] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:41.389241 12973 raft_consensus.cc:1261] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:41.389886 19731 consensus_queue.cc:1035] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:41.403199 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:41.432222 19762 mvcc.cc:205] Tried to move back new op lower bound from 7022801557002350592 to 7022801555864260608. Current Snapshot: MvccSnapshot[applied={T|T < 7022801557002350592}]
I20240501 08:12:41.434294 19772 tablet.cc:1722] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:41.437345 19772 tablet.cc:1620] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: MemRowSet was empty: no flush needed.
I20240501 08:12:41.457765 12827 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 (table rebalancer_dynamic_table [id=389d409bbeec43a28e75d26b8d520887]). Expected version 1 got 0
I20240501 08:12:41.456745 19773 tablet.cc:1722] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:41.459178 19773 tablet.cc:1620] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:41.478498 12827 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=389d409bbeec43a28e75d26b8d520887] alter complete (version 1)
I20240501 08:12:41.531723 19774 tablet.cc:1722] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:41.534744 19774 tablet.cc:1620] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:41.534768 12827 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 (table rebalancer_dynamic_table [id=389d409bbeec43a28e75d26b8d520887]). Expected version 1 got 0
I20240501 08:12:41.557256 12827 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:41.583081 13084 tablet_service.cc:1467] Processing CreateTablet for tablet c99eddd74b7740dcab6b93ca437b51ae (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:41.583568 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99eddd74b7740dcab6b93ca437b51ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.590318 15870 tablet_service.cc:1467] Processing CreateTablet for tablet c99eddd74b7740dcab6b93ca437b51ae (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:41.591841 16004 tablet_service.cc:1467] Processing CreateTablet for tablet f755002e2f824db7993f912d234295bb (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:41.592447 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f755002e2f824db7993f912d234295bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.596946 16138 tablet_service.cc:1467] Processing CreateTablet for tablet f755002e2f824db7993f912d234295bb (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:41.597417 16138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f755002e2f824db7993f912d234295bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.598330 13213 tablet_service.cc:1467] Processing CreateTablet for tablet c99eddd74b7740dcab6b93ca437b51ae (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:41.598719 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99eddd74b7740dcab6b93ca437b51ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.600257 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 08106d7d42674f01bbee94b6236fb9f4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:41.600617 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08106d7d42674f01bbee94b6236fb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.600462 12827 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:41.601444 13085 tablet_service.cc:1467] Processing CreateTablet for tablet f755002e2f824db7993f912d234295bb (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:41.601820 13085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f755002e2f824db7993f912d234295bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.605144 13083 tablet_service.cc:1467] Processing CreateTablet for tablet 08106d7d42674f01bbee94b6236fb9f4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:41.605487 13083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08106d7d42674f01bbee94b6236fb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.610374 13343 tablet_service.cc:1467] Processing CreateTablet for tablet 08106d7d42674f01bbee94b6236fb9f4 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:41.610757 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08106d7d42674f01bbee94b6236fb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.618420 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99eddd74b7740dcab6b93ca437b51ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:41.625633 19778 tablet_bootstrap.cc:492] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:41.626482 19777 tablet_bootstrap.cc:492] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:41.628671 19777 tablet_bootstrap.cc:654] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:41.629698 19778 tablet_bootstrap.cc:654] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:41.631323 19776 tablet_bootstrap.cc:492] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:41.639688 19777 tablet_bootstrap.cc:492] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb: No bootstrap required, opened a new log
I20240501 08:12:41.640458 19772 tablet.cc:1722] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:41.640921 19772 tablet.cc:1620] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: MemRowSet was empty: no flush needed.
I20240501 08:12:41.639941 19777 ts_tablet_manager.cc:1367] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240501 08:12:41.641963 19777 raft_consensus.cc:348] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:41.642306 19777 raft_consensus.cc:374] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:41.642439 19777 raft_consensus.cc:729] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:41.645229 19773 tablet.cc:1722] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:41.645197 19777 consensus_queue.cc:260] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:41.645712 19773 tablet.cc:1620] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:41.646234 19777 ts_tablet_manager.cc:1398] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240501 08:12:41.658185 19776 tablet_bootstrap.cc:654] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:41.660708 12830 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=389d409bbeec43a28e75d26b8d520887] alter complete (version 2)
I20240501 08:12:41.674839 19776 tablet_bootstrap.cc:492] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:41.675091 19776 ts_tablet_manager.cc:1367] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.000s
I20240501 08:12:41.675894 19776 raft_consensus.cc:348] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:41.676317 19776 raft_consensus.cc:374] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:41.676545 19776 raft_consensus.cc:729] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:41.676993 19776 consensus_queue.cc:260] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:41.677950 19776 ts_tablet_manager.cc:1398] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:41.699518 19774 tablet.cc:1722] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:41.700071 19774 tablet.cc:1620] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:41.729050 19783 tablet_bootstrap.cc:492] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:41.731693 19776 tablet_bootstrap.cc:492] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:41.741199 19783 tablet_bootstrap.cc:654] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:41.741910 19782 tablet_bootstrap.cc:492] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:41.745362 19784 tablet_bootstrap.cc:492] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:41.751569 19784 tablet_bootstrap.cc:654] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:41.744625 19782 tablet_bootstrap.cc:654] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:41.755072 19778 tablet_bootstrap.cc:492] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:41.755214 19778 ts_tablet_manager.cc:1367] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.130s user 0.011s sys 0.055s
I20240501 08:12:41.756057 19778 raft_consensus.cc:348] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:41.756327 19778 raft_consensus.cc:374] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:41.756387 19778 raft_consensus.cc:729] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:41.756633 19778 consensus_queue.cc:260] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:41.761526 19778 ts_tablet_manager.cc:1398] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:41.745990 19732 raft_consensus.cc:482] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:41.766628 19732 raft_consensus.cc:504] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:41.767819 19732 leader_election.cc:288] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:41.769410 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08106d7d42674f01bbee94b6236fb9f4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:41.769655 13363 raft_consensus.cc:2454] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 0.
I20240501 08:12:41.770192 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08106d7d42674f01bbee94b6236fb9f4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:41.770543 19776 tablet_bootstrap.cc:654] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
W20240501 08:12:41.770859 13170 leader_election.cc:341] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:41.771430 13170 leader_election.cc:302] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters: e3fefd4dca2d41678187047f06c10187
I20240501 08:12:41.771839 19732 raft_consensus.cc:2781] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:41.771963 19732 raft_consensus.cc:482] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:41.772047 19732 raft_consensus.cc:3037] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:41.772811 12827 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:41.775794 19732 raft_consensus.cc:504] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:41.782642 19783 tablet_bootstrap.cc:492] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:41.782826 19783 ts_tablet_manager.cc:1367] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.000s
I20240501 08:12:41.786900 19783 raft_consensus.cc:348] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:41.787087 19783 raft_consensus.cc:374] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:41.787156 19783 raft_consensus.cc:729] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:41.787403 19783 consensus_queue.cc:260] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:41.792002 19783 ts_tablet_manager.cc:1398] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240501 08:12:41.797129 19772 tablet.cc:1722] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:41.803296 19774 tablet.cc:1722] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:41.804435 19773 tablet.cc:1722] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:41.830901 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08106d7d42674f01bbee94b6236fb9f4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:41.831136 13363 raft_consensus.cc:3037] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:41.831331 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=389d409bbeec43a28e75d26b8d520887] alter complete (version 3)
I20240501 08:12:41.834017 13363 raft_consensus.cc:2454] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:12:41.835093 13170 leader_election.cc:302] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:41.836432 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08106d7d42674f01bbee94b6236fb9f4" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
W20240501 08:12:41.837144 13170 leader_election.cc:341] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:41.837525 19732 leader_election.cc:288] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:41.837646 19730 raft_consensus.cc:2781] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:41.837852 19730 raft_consensus.cc:686] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:41.838199 19730 consensus_queue.cc:237] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:41.851416 19784 tablet_bootstrap.cc:492] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:41.851629 19784 ts_tablet_manager.cc:1367] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.106s user 0.015s sys 0.015s
I20240501 08:12:41.853207 19784 raft_consensus.cc:348] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:41.853505 19784 raft_consensus.cc:374] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:41.853600 19784 raft_consensus.cc:729] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:41.853930 19784 consensus_queue.cc:260] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:41.855170 19782 tablet_bootstrap.cc:492] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:41.855326 19782 ts_tablet_manager.cc:1367] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.114s user 0.007s sys 0.023s
I20240501 08:12:41.856011 19782 raft_consensus.cc:348] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:41.856253 19782 raft_consensus.cc:374] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:41.856336 19782 raft_consensus.cc:729] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:41.856603 19782 consensus_queue.cc:260] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:41.859330 19782 ts_tablet_manager.cc:1398] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:12:41.859617 19782 tablet_bootstrap.cc:492] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:41.868391 12830 catalog_manager.cc:5484] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:41.879453 19776 tablet_bootstrap.cc:492] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:41.879611 19776 ts_tablet_manager.cc:1367] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.148s user 0.009s sys 0.009s
I20240501 08:12:41.880281 19776 raft_consensus.cc:348] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:41.880508 19776 raft_consensus.cc:374] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:41.880559 19776 raft_consensus.cc:729] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:41.881342 19776 consensus_queue.cc:260] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:41.882025 19776 ts_tablet_manager.cc:1398] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:12:41.883219 19784 ts_tablet_manager.cc:1398] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.031s user 0.000s sys 0.002s
I20240501 08:12:41.884712 19782 tablet_bootstrap.cc:654] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:41.897905 19782 tablet_bootstrap.cc:492] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:41.898074 19782 ts_tablet_manager.cc:1367] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.002s
I20240501 08:12:41.898823 19782 raft_consensus.cc:348] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:41.899045 19782 raft_consensus.cc:374] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:41.899102 19782 raft_consensus.cc:729] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:41.899353 19782 consensus_queue.cc:260] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:41.903970 19782 ts_tablet_manager.cc:1398] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240501 08:12:41.908262 19782 tablet_bootstrap.cc:492] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:41.910210 19782 tablet_bootstrap.cc:654] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:41.932895 19782 tablet_bootstrap.cc:492] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:41.934022 19782 ts_tablet_manager.cc:1367] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.016s
I20240501 08:12:41.934731 19782 raft_consensus.cc:348] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:41.934947 19782 raft_consensus.cc:374] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:41.935060 19782 raft_consensus.cc:729] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:41.935334 19782 consensus_queue.cc:260] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:41.947290 19782 ts_tablet_manager.cc:1398] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240501 08:12:41.948235 12830 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:41.949576 12830 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:41.959933 19663 raft_consensus.cc:482] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:41.960148 19663 raft_consensus.cc:504] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:41.960775 19663 leader_election.cc:288] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:41.962098 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f755002e2f824db7993f912d234295bb" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:12:41.962354 16153 raft_consensus.cc:2454] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:41.963996 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f755002e2f824db7993f912d234295bb" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:41.964195 13104 raft_consensus.cc:2454] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:41.966993 15957 leader_election.cc:302] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:12:41.967729 19477 raft_consensus.cc:2781] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:41.967783 12830 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0
I20240501 08:12:41.967854 19477 raft_consensus.cc:482] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:41.967947 19477 raft_consensus.cc:3037] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:41.968717 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:41 UTC) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:41.969238 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:41 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:41.981652 16004 tablet_service.cc:1514] Processing DeleteTablet for tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:41 UTC) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:42.012639 19806 tablet_replica.cc:330] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:42.012979 19806 raft_consensus.cc:2229] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.013283 19806 raft_consensus.cc:2258] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.013857 19807 tablet_replica.cc:330] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:42.014133 19807 raft_consensus.cc:2229] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.014387 19807 raft_consensus.cc:2258] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.015312 19807 ts_tablet_manager.cc:1875] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:42.015739 19805 tablet_replica.cc:330] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:42.017390 19805 raft_consensus.cc:2229] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:42.017812 19805 raft_consensus.cc:2258] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.021301 19805 ts_tablet_manager.cc:1875] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:42.031111 19805 ts_tablet_manager.cc:1888] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:42.035038 19807 ts_tablet_manager.cc:1888] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:42.035190 19807 log.cc:1196] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/80fda9fa3ddf411c8efb1d82cfa7c0f0
I20240501 08:12:42.035668 19807 ts_tablet_manager.cc:1909] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 5dd722c770d34d6c9154a87917117161: Deleting consensus metadata
I20240501 08:12:42.035827 19477 raft_consensus.cc:504] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:42.036710 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 (table rebalancer_dynamic_table_new_name [id=389d409bbeec43a28e75d26b8d520887]) successfully deleted
I20240501 08:12:42.037001 19805 log.cc:1196] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/80fda9fa3ddf411c8efb1d82cfa7c0f0
I20240501 08:12:42.037461 19805 ts_tablet_manager.cc:1909] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 6ed448d04e4646658c0de37d829ff15b: Deleting consensus metadata
I20240501 08:12:42.038388 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f755002e2f824db7993f912d234295bb" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:42.038455 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f755002e2f824db7993f912d234295bb" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:12:42.038594 13104 raft_consensus.cc:3037] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:42.038669 16157 raft_consensus.cc:3037] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:42.041409 13104 raft_consensus.cc:2454] T f755002e2f824db7993f912d234295bb P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:42.041536 16157 raft_consensus.cc:2454] T f755002e2f824db7993f912d234295bb P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:42.041941 15959 leader_election.cc:302] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:42.042272 19404 raft_consensus.cc:2781] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:42.042423 19404 raft_consensus.cc:686] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:42.042727 19404 consensus_queue.cc:237] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:42.043825 19477 leader_election.cc:288] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:12:42.046591 19806 ts_tablet_manager.cc:1875] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:42.048491 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 (table rebalancer_dynamic_table_new_name [id=389d409bbeec43a28e75d26b8d520887]) successfully deleted
I20240501 08:12:42.056999 12830 catalog_manager.cc:5484] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:42.075510 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:42.081306 19806 ts_tablet_manager.cc:1888] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:42.081539 19806 log.cc:1196] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/80fda9fa3ddf411c8efb1d82cfa7c0f0
I20240501 08:12:42.082039 19806 ts_tablet_manager.cc:1909] T 80fda9fa3ddf411c8efb1d82cfa7c0f0 P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:42.083217 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 80fda9fa3ddf411c8efb1d82cfa7c0f0 (table rebalancer_dynamic_table_new_name [id=389d409bbeec43a28e75d26b8d520887]) successfully deleted
I20240501 08:12:42.088814 13343 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab9f992762d4b28b489020dc39b49ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=15fbbdedf6c4433891821c797e0dd8d7]), partition=
I20240501 08:12:42.089206 19480 raft_consensus.cc:482] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:42.089331 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab9f992762d4b28b489020dc39b49ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.089346 19480 raft_consensus.cc:504] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:42.090165 19480 leader_election.cc:288] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:42.091209 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99eddd74b7740dcab6b93ca437b51ae" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:42.091425 15891 raft_consensus.cc:2454] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:42.091809 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99eddd74b7740dcab6b93ca437b51ae" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:42.092003 13232 raft_consensus.cc:2454] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:42.093053 13041 leader_election.cc:302] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:42.097199 12953 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab9f992762d4b28b489020dc39b49ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=15fbbdedf6c4433891821c797e0dd8d7]), partition=
I20240501 08:12:42.097582 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab9f992762d4b28b489020dc39b49ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.098484 19480 raft_consensus.cc:2781] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:42.098579 19480 raft_consensus.cc:482] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:42.098651 19480 raft_consensus.cc:3037] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:42.100122 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 7ab9f992762d4b28b489020dc39b49ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=15fbbdedf6c4433891821c797e0dd8d7]), partition=
I20240501 08:12:42.100554 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ab9f992762d4b28b489020dc39b49ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.101279 19480 raft_consensus.cc:504] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:42.104449 19778 tablet_bootstrap.cc:492] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:42.105733 19778 tablet_bootstrap.cc:654] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:42.109004 19783 tablet_bootstrap.cc:492] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:42.110677 19783 tablet_bootstrap.cc:654] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:42.114215 15891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99eddd74b7740dcab6b93ca437b51ae" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:42.114441 15891 raft_consensus.cc:3037] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:42.119104 15891 raft_consensus.cc:2454] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:42.119568 13038 leader_election.cc:302] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:42.119926 19624 raft_consensus.cc:2781] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:42.122794 19783 tablet_bootstrap.cc:492] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:42.122704 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99eddd74b7740dcab6b93ca437b51ae" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:42.122984 19783 ts_tablet_manager.cc:1367] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20240501 08:12:42.122983 13232 raft_consensus.cc:3037] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:42.123941 19783 raft_consensus.cc:348] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:42.124229 19783 raft_consensus.cc:374] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:42.124423 19783 raft_consensus.cc:729] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:42.124680 19783 consensus_queue.cc:260] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:42.126785 19783 ts_tablet_manager.cc:1398] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 08:12:42.159253 13232 raft_consensus.cc:2454] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:42.160060 19480 leader_election.cc:288] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:42.184547 19820 raft_consensus.cc:482] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:42.184720 19820 raft_consensus.cc:504] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:42.185505 19820 leader_election.cc:288] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:42.185961 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08106d7d42674f01bbee94b6236fb9f4" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:42.185942 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08106d7d42674f01bbee94b6236fb9f4" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:42.186203 13363 raft_consensus.cc:2379] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3fefd4dca2d41678187047f06c10187 in current term 1: Already voted for candidate 6ed448d04e4646658c0de37d829ff15b in this term.
I20240501 08:12:42.186337 19624 raft_consensus.cc:686] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:42.186756 13040 leader_election.cc:302] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3fefd4dca2d41678187047f06c10187; no voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:42.186719 19624 consensus_queue.cc:237] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:42.187163 19820 raft_consensus.cc:3037] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:42.190358 19778 tablet_bootstrap.cc:492] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:42.190410 19820 raft_consensus.cc:2726] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:42.190601 19778 ts_tablet_manager.cc:1367] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.086s user 0.002s sys 0.019s
I20240501 08:12:42.191107 19812 tablet_bootstrap.cc:492] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:42.192963 19812 tablet_bootstrap.cc:654] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:42.193676 19778 raft_consensus.cc:348] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:42.193987 19778 raft_consensus.cc:374] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:42.194100 19778 raft_consensus.cc:729] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:42.194347 12830 catalog_manager.cc:5484] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:42.195788 19778 consensus_queue.cc:260] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:42.199764 19778 ts_tablet_manager.cc:1398] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20240501 08:12:42.201953 19809 raft_consensus.cc:482] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:42.202122 19809 raft_consensus.cc:504] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:42.203028 19809 leader_election.cc:288] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:42.203383 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab9f992762d4b28b489020dc39b49ac" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:42.203562 13363 raft_consensus.cc:2454] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:42.203596 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab9f992762d4b28b489020dc39b49ac" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
W20240501 08:12:42.204162 15826 leader_election.cc:341] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:42.204344 15825 leader_election.cc:302] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:42.204816 19809 raft_consensus.cc:2781] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:42.204928 19809 raft_consensus.cc:482] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:42.205001 19809 raft_consensus.cc:3037] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:42.210413 19809 raft_consensus.cc:504] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:42.211855 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab9f992762d4b28b489020dc39b49ac" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:42.212041 13363 raft_consensus.cc:3037] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:42.213999 13232 consensus_queue.cc:237] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:42.214709 13363 raft_consensus.cc:2454] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:42.215185 15825 leader_election.cc:302] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters:
I20240501 08:12:42.215512 19436 raft_consensus.cc:2781] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:42.215647 19436 raft_consensus.cc:686] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:42.215888 19436 consensus_queue.cc:237] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:42.222947 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ab9f992762d4b28b489020dc39b49ac" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
W20240501 08:12:42.223632 15826 leader_election.cc:341] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:42.223762 16153 raft_consensus.cc:1261] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:42.224347 19730 consensus_queue.cc:1035] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:42.225068 19809 leader_election.cc:288] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:42.223960 13363 raft_consensus.cc:1261] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:42.225693 19730 consensus_queue.cc:1035] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:42.231709 19812 tablet_bootstrap.cc:492] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:42.232627 19812 ts_tablet_manager.cc:1367] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20240501 08:12:42.235026 19812 raft_consensus.cc:348] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:42.235226 19812 raft_consensus.cc:374] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:42.235268 19812 raft_consensus.cc:729] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:42.236305 19812 consensus_queue.cc:260] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:42.244913 12830 catalog_manager.cc:5484] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:42.246781 19730 raft_consensus.cc:2932] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:42.250373 19812 ts_tablet_manager.cc:1398] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240501 08:12:42.252504 13363 raft_consensus.cc:2932] T 7883772f69cd4a18a2a6d568adbdeb61 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:42.259130 12830 catalog_manager.cc:5484] T 7883772f69cd4a18a2a6d568adbdeb61 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:42.264526 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 7883772f69cd4a18a2a6d568adbdeb61 with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:42.264866 19806 tablet_replica.cc:330] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:42.265035 19806 raft_consensus.cc:2229] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.265223 19806 raft_consensus.cc:2258] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.265959 19806 ts_tablet_manager.cc:1875] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:42.269857 16153 raft_consensus.cc:2932] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:42.273829 19806 ts_tablet_manager.cc:1888] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:42.274037 19806 log.cc:1196] T 7883772f69cd4a18a2a6d568adbdeb61 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/7883772f69cd4a18a2a6d568adbdeb61
I20240501 08:12:42.275214 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 7883772f69cd4a18a2a6d568adbdeb61 (table rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]) successfully deleted
I20240501 08:12:42.351404 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:42.370564 13364 raft_consensus.cc:1261] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:42.371233 12974 raft_consensus.cc:3037] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:42.371279 19436 consensus_queue.cc:1035] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:42.374570 12974 raft_consensus.cc:1261] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:42.376904 19436 consensus_queue.cc:1035] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:42.405953 12830 catalog_manager.cc:5576] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839) does not have the latest schema for tablet 7ab9f992762d4b28b489020dc39b49ac (table rebalancer_dynamic_table [id=15fbbdedf6c4433891821c797e0dd8d7]). Expected version 1 got 0
I20240501 08:12:42.419694 19862 tablet.cc:1722] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:42.420323 19862 tablet.cc:1620] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:42.422644 12830 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet 7ab9f992762d4b28b489020dc39b49ac (table rebalancer_dynamic_table [id=15fbbdedf6c4433891821c797e0dd8d7]). Expected version 1 got 0
I20240501 08:12:42.424422 12831 catalog_manager.cc:5576] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839) does not have the latest schema for tablet 7ab9f992762d4b28b489020dc39b49ac (table rebalancer_dynamic_table [id=15fbbdedf6c4433891821c797e0dd8d7]). Expected version 1 got 0
I20240501 08:12:42.434326 12831 catalog_manager.cc:5576] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991) does not have the latest schema for tablet 7ab9f992762d4b28b489020dc39b49ac (table rebalancer_dynamic_table [id=15fbbdedf6c4433891821c797e0dd8d7]). Expected version 1 got 0
I20240501 08:12:42.435127 19732 consensus_queue.cc:1035] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:42.438999 19732 consensus_queue.cc:1035] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:42.439386 19861 tablet.cc:1722] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:42.440711 19865 tablet.cc:1722] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:42.441397 19865 tablet.cc:1620] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:42.442492 19861 tablet.cc:1620] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:42.495577 12830 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=15fbbdedf6c4433891821c797e0dd8d7] alter complete (version 1)
I20240501 08:12:42.522392 19404 consensus_queue.cc:1035] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:42.606576 12826 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240501 08:12:42.624258 19861 tablet.cc:1722] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:42.624876 19861 tablet.cc:1620] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:42.630785 19865 tablet.cc:1722] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:42.631109 19862 tablet.cc:1722] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240501 08:12:42.631342 19865 tablet.cc:1620] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:12:42.631542 19862 tablet.cc:1620] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: MemRowSet was empty: no flush needed.
I20240501 08:12:42.639084 19663 consensus_queue.cc:1035] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:42.650528 12831 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=15fbbdedf6c4433891821c797e0dd8d7] alter complete (version 2)
I20240501 08:12:42.671954 19624 consensus_queue.cc:1035] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:42.688608 19815 consensus_queue.cc:1035] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:42.760087 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240501 08:12:42.769678 19861 tablet.cc:1722] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:42.770483 19865 tablet.cc:1722] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:42.772423 19862 tablet.cc:1722] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240501 08:12:42.782869 12830 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table_new_name [id=15fbbdedf6c4433891821c797e0dd8d7] alter complete (version 3)
I20240501 08:12:42.825358 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:12:42.838260 16004 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:42.839202 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:12:42.854770 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:42.854905 13343 tablet_service.cc:1467] Processing CreateTablet for tablet 6658ae0ccbcc4baf8853ab8778049030 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:42.855348 13343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6658ae0ccbcc4baf8853ab8778049030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.856861 13084 tablet_service.cc:1467] Processing CreateTablet for tablet da0cf43235be4fa9bf53b1067138bc72 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:42.857211 13084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0cf43235be4fa9bf53b1067138bc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.860913 12953 tablet_service.cc:1467] Processing CreateTablet for tablet da0cf43235be4fa9bf53b1067138bc72 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:42.861348 12953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0cf43235be4fa9bf53b1067138bc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.861377 15736 tablet_service.cc:1467] Processing CreateTablet for tablet da0cf43235be4fa9bf53b1067138bc72 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:42.861855 15736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da0cf43235be4fa9bf53b1067138bc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.869657 13213 tablet_service.cc:1467] Processing CreateTablet for tablet 6658ae0ccbcc4baf8853ab8778049030 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:42.870138 13213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6658ae0ccbcc4baf8853ab8778049030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.873989 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 236d2386bb2d4f378faeabce22238402 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:42.873970 13215 tablet_service.cc:1467] Processing CreateTablet for tablet 236d2386bb2d4f378faeabce22238402 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:42.874361 13215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236d2386bb2d4f378faeabce22238402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.874397 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236d2386bb2d4f378faeabce22238402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.892346 12830 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:42.892591 12830 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240501 08:12:42.896855 12830 catalog_manager.cc:5771] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: Sending DeleteTablet for 3 replicas of tablet 7ab9f992762d4b28b489020dc39b49ac
I20240501 08:12:42.898059 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab9f992762d4b28b489020dc39b49ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:42 UTC) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:42.898474 13344 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab9f992762d4b28b489020dc39b49ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:42 UTC) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:42.899693 12954 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ab9f992762d4b28b489020dc39b49ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-01 08:12:42 UTC) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:42.900403 16004 tablet_service.cc:1467] Processing CreateTablet for tablet 6658ae0ccbcc4baf8853ab8778049030 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:42.901046 16004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6658ae0ccbcc4baf8853ab8778049030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.912199 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 236d2386bb2d4f378faeabce22238402 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:12:42.912742 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236d2386bb2d4f378faeabce22238402. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:42.911070 19913 tablet_bootstrap.cc:492] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:42.917009 19920 tablet_replica.cc:330] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:42.918274 19920 raft_consensus.cc:2229] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.919025 19920 raft_consensus.cc:2258] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.924978 19920 ts_tablet_manager.cc:1875] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:42.929531 15871 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:42.939749 19913 tablet_bootstrap.cc:654] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:42.943078 19920 ts_tablet_manager.cc:1888] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:42.956274 19920 log.cc:1196] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/7ab9f992762d4b28b489020dc39b49ac
I20240501 08:12:42.956825 19920 ts_tablet_manager.cc:1909] T 7ab9f992762d4b28b489020dc39b49ac P 4ecb724cb4764a53988fde00407db4c0: Deleting consensus metadata
I20240501 08:12:42.958057 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 7ab9f992762d4b28b489020dc39b49ac (table rebalancer_dynamic_table_new_name [id=15fbbdedf6c4433891821c797e0dd8d7]) successfully deleted
I20240501 08:12:42.964648 19915 tablet_bootstrap.cc:492] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:42.969745 19921 tablet_replica.cc:330] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:42.970037 19921 raft_consensus.cc:2229] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:42.970245 19921 raft_consensus.cc:2258] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.971318 19921 ts_tablet_manager.cc:1875] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:42.977540 19918 tablet_bootstrap.cc:492] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:42.984853 19919 tablet_replica.cc:330] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:42.985272 19919 raft_consensus.cc:2229] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Raft consensus shutting down.
I20240501 08:12:42.985859 19919 raft_consensus.cc:2258] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:42.989876 19919 ts_tablet_manager.cc:1875] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240501 08:12:43.001014 12830 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46968:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240501 08:12:43.002278 19922 tablet_bootstrap.cc:492] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:43.007422 19915 tablet_bootstrap.cc:654] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.010848 15733 tablet_service.cc:1467] Processing CreateTablet for tablet 66af91fdcce84b779cf6423562b582b0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b6b6af5609ce409ba8be11e257f107ef]), partition=
I20240501 08:12:43.011058 15872 tablet_service.cc:1467] Processing CreateTablet for tablet 66af91fdcce84b779cf6423562b582b0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b6b6af5609ce409ba8be11e257f107ef]), partition=
I20240501 08:12:43.011257 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66af91fdcce84b779cf6423562b582b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:43.011461 15872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66af91fdcce84b779cf6423562b582b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:43.011672 19916 tablet_bootstrap.cc:492] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:43.016233 19913 tablet_bootstrap.cc:492] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0: No bootstrap required, opened a new log
I20240501 08:12:43.016419 19913 ts_tablet_manager.cc:1367] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.105s user 0.002s sys 0.008s
I20240501 08:12:43.016558 16005 tablet_service.cc:1467] Processing CreateTablet for tablet 66af91fdcce84b779cf6423562b582b0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b6b6af5609ce409ba8be11e257f107ef]), partition=
I20240501 08:12:43.017236 19913 raft_consensus.cc:348] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.017494 19913 raft_consensus.cc:374] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.017591 19913 raft_consensus.cc:729] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.017818 16005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66af91fdcce84b779cf6423562b582b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:43.017879 19913 consensus_queue.cc:260] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.023531 19913 ts_tablet_manager.cc:1398] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240501 08:12:43.027606 19917 tablet_bootstrap.cc:492] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:43.040364 19916 tablet_bootstrap.cc:654] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.043277 19917 tablet_bootstrap.cc:654] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.045702 19918 tablet_bootstrap.cc:654] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.062548 19916 tablet_bootstrap.cc:492] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2: No bootstrap required, opened a new log
I20240501 08:12:43.062705 19916 ts_tablet_manager.cc:1367] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20240501 08:12:43.065912 19922 tablet_bootstrap.cc:654] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.063457 19916 raft_consensus.cc:348] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.071041 19916 raft_consensus.cc:374] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.071116 19916 raft_consensus.cc:729] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.073890 19916 consensus_queue.cc:260] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.077841 19916 ts_tablet_manager.cc:1398] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240501 08:12:43.083513 19914 tablet_bootstrap.cc:492] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:43.088800 19915 tablet_bootstrap.cc:492] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:43.088937 19915 ts_tablet_manager.cc:1367] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.124s user 0.000s sys 0.004s
I20240501 08:12:43.089589 19915 raft_consensus.cc:348] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.089797 19915 raft_consensus.cc:374] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.089848 19915 raft_consensus.cc:729] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.090090 19915 consensus_queue.cc:260] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.090716 19915 ts_tablet_manager.cc:1398] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:43.091157 19915 tablet_bootstrap.cc:492] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:43.092764 19917 tablet_bootstrap.cc:492] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187: No bootstrap required, opened a new log
I20240501 08:12:43.092957 19917 ts_tablet_manager.cc:1367] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20240501 08:12:43.093753 19917 raft_consensus.cc:348] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.094511 19917 raft_consensus.cc:374] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.094636 19917 raft_consensus.cc:729] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.094945 19917 consensus_queue.cc:260] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.095445 19915 tablet_bootstrap.cc:654] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.097230 19917 ts_tablet_manager.cc:1398] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:12:43.105852 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:43.120579 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:43.134003 19918 tablet_bootstrap.cc:492] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:43.134197 19918 ts_tablet_manager.cc:1367] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.157s user 0.005s sys 0.000s
I20240501 08:12:43.139576 19914 tablet_bootstrap.cc:654] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.149273 19918 raft_consensus.cc:348] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:43.149567 19918 raft_consensus.cc:374] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.149647 19918 raft_consensus.cc:729] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.149854 19918 consensus_queue.cc:260] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:43.156350 19918 ts_tablet_manager.cc:1398] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240501 08:12:43.157259 19922 tablet_bootstrap.cc:492] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:43.160199 19922 ts_tablet_manager.cc:1367] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.155s user 0.004s sys 0.000s
I20240501 08:12:43.161336 19789 raft_consensus.cc:482] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:43.161499 19789 raft_consensus.cc:504] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.161913 19922 raft_consensus.cc:348] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:43.162276 19789 leader_election.cc:288] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:43.162462 19922 raft_consensus.cc:374] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.162931 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6658ae0ccbcc4baf8853ab8778049030" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:43.163004 19922 raft_consensus.cc:729] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.163277 19922 consensus_queue.cc:260] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:43.163311 13235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6658ae0ccbcc4baf8853ab8778049030" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:43.163583 13235 raft_consensus.cc:2454] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 0.
W20240501 08:12:43.163687 13300 leader_election.cc:341] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:43.163844 19922 ts_tablet_manager.cc:1398] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:43.163995 13301 leader_election.cc:302] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters: 5dd722c770d34d6c9154a87917117161
I20240501 08:12:43.164170 19922 tablet_bootstrap.cc:492] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:43.164404 19789 raft_consensus.cc:2781] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:43.164500 19789 raft_consensus.cc:482] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:43.164588 19789 raft_consensus.cc:3037] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.166486 19922 tablet_bootstrap.cc:654] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.167255 19789 raft_consensus.cc:504] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.168329 19789 leader_election.cc:288] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:43.168550 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6658ae0ccbcc4baf8853ab8778049030" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:43.168751 13232 raft_consensus.cc:3037] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.168757 19918 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:43.171303 13232 raft_consensus.cc:2454] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:12:43.171845 13301 leader_election.cc:302] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:43.172699 19789 raft_consensus.cc:2781] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:43.172889 19789 raft_consensus.cc:686] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:12:43.173210 19789 consensus_queue.cc:237] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.174286 13084 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:43.187331 19815 raft_consensus.cc:482] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:43.187490 19815 raft_consensus.cc:504] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.188225 19815 leader_election.cc:288] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:43.188949 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0cf43235be4fa9bf53b1067138bc72" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:43.189170 12971 raft_consensus.cc:2454] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 0.
I20240501 08:12:43.189702 13041 leader_election.cc:302] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:43.190021 19815 raft_consensus.cc:2781] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:43.190085 19815 raft_consensus.cc:482] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:43.190136 19815 raft_consensus.cc:3037] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.192893 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0cf43235be4fa9bf53b1067138bc72" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:43.193017 19815 raft_consensus.cc:504] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.194286 19815 leader_election.cc:288] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:43.195549 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0cf43235be4fa9bf53b1067138bc72" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:12:43.195763 12971 raft_consensus.cc:3037] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.197834 12971 raft_consensus.cc:2454] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
I20240501 08:12:43.205057 13041 leader_election.cc:302] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:12:43.205415 19815 raft_consensus.cc:2781] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:43.205524 19815 raft_consensus.cc:686] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:12:43.205801 19815 consensus_queue.cc:237] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.207882 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0cf43235be4fa9bf53b1067138bc72" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
W20240501 08:12:43.210147 13040 leader_election.cc:341] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
W20240501 08:12:43.210481 13040 leader_election.cc:341] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:43.211978 19921 ts_tablet_manager.cc:1888] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:43.212121 19921 log.cc:1196] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/7ab9f992762d4b28b489020dc39b49ac
I20240501 08:12:43.212604 19921 ts_tablet_manager.cc:1909] T 7ab9f992762d4b28b489020dc39b49ac P 76d830e199eb461c8ca689c9460ee0c2: Deleting consensus metadata
I20240501 08:12:43.214037 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 7ab9f992762d4b28b489020dc39b49ac (table rebalancer_dynamic_table_new_name [id=15fbbdedf6c4433891821c797e0dd8d7]) successfully deleted
I20240501 08:12:43.219240 19914 tablet_bootstrap.cc:492] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:43.226155 19914 ts_tablet_manager.cc:1367] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.158s user 0.002s sys 0.002s
I20240501 08:12:43.226935 12830 catalog_manager.cc:5484] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:43.227052 19914 raft_consensus.cc:348] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.227262 19914 raft_consensus.cc:374] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.227342 19914 raft_consensus.cc:729] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.227620 19914 consensus_queue.cc:260] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.228346 19914 ts_tablet_manager.cc:1398] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:43.232733 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:43.241667 12833 catalog_manager.cc:5484] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:43.243978 19918 tablet_bootstrap.cc:654] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.259421 19915 tablet_bootstrap.cc:492] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b: No bootstrap required, opened a new log
I20240501 08:12:43.259598 19915 ts_tablet_manager.cc:1367] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.168s user 0.001s sys 0.009s
I20240501 08:12:43.260321 19915 raft_consensus.cc:348] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:43.261744 19915 raft_consensus.cc:374] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.261853 19915 raft_consensus.cc:729] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.262089 19915 consensus_queue.cc:260] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:43.264169 19914 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:12:43.264856 19915 ts_tablet_manager.cc:1398] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:43.266664 19914 tablet_bootstrap.cc:654] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.274801 19914 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: No bootstrap required, opened a new log
I20240501 08:12:43.274993 19914 ts_tablet_manager.cc:1367] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240501 08:12:43.276163 19914 raft_consensus.cc:348] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.276398 19914 raft_consensus.cc:374] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.276494 19914 raft_consensus.cc:729] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.276875 19914 consensus_queue.cc:260] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.282562 19914 ts_tablet_manager.cc:1398] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 08:12:43.283872 19919 ts_tablet_manager.cc:1888] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240501 08:12:43.284011 19919 log.cc:1196] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/7ab9f992762d4b28b489020dc39b49ac
I20240501 08:12:43.288089 19919 ts_tablet_manager.cc:1909] T 7ab9f992762d4b28b489020dc39b49ac P 4d516916eeac468db514eeb928f06190: Deleting consensus metadata
I20240501 08:12:43.169386 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6658ae0ccbcc4baf8853ab8778049030" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:12:43.278509 19663 raft_consensus.cc:482] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:43.288487 19663 raft_consensus.cc:504] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:43.289254 19663 leader_election.cc:288] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:12:43.290208 13300 leader_election.cc:341] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:43.290796 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236d2386bb2d4f378faeabce22238402" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:43.291023 13232 raft_consensus.cc:2454] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:43.292055 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 7ab9f992762d4b28b489020dc39b49ac (table rebalancer_dynamic_table_new_name [id=15fbbdedf6c4433891821c797e0dd8d7]) successfully deleted
I20240501 08:12:43.293182 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236d2386bb2d4f378faeabce22238402" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:43.293439 15892 raft_consensus.cc:2454] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 0.
I20240501 08:12:43.293831 15960 leader_election.cc:302] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:43.294226 19918 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190: No bootstrap required, opened a new log
I20240501 08:12:43.294405 19918 ts_tablet_manager.cc:1367] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.126s user 0.004s sys 0.008s
I20240501 08:12:43.294406 19404 raft_consensus.cc:2781] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:43.294507 19404 raft_consensus.cc:482] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:43.294603 19404 raft_consensus.cc:3037] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.295079 19918 raft_consensus.cc:348] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.295253 19918 raft_consensus.cc:374] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.295336 19918 raft_consensus.cc:729] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.295568 19918 consensus_queue.cc:260] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.297474 19404 raft_consensus.cc:504] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:43.301227 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236d2386bb2d4f378faeabce22238402" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:12:43.301436 15892 raft_consensus.cc:3037] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.301596 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236d2386bb2d4f378faeabce22238402" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:43.301767 13232 raft_consensus.cc:3037] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.304097 15892 raft_consensus.cc:2454] T 236d2386bb2d4f378faeabce22238402 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:43.304908 15957 leader_election.cc:302] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 5dd722c770d34d6c9154a87917117161; no voters:
I20240501 08:12:43.305262 19663 raft_consensus.cc:2781] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:43.305397 19663 raft_consensus.cc:686] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:43.305694 19663 consensus_queue.cc:237] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:43.308665 19918 ts_tablet_manager.cc:1398] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240501 08:12:43.309100 19404 leader_election.cc:288] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:43.310225 19922 tablet_bootstrap.cc:492] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:43.310397 19922 ts_tablet_manager.cc:1367] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.146s user 0.004s sys 0.000s
I20240501 08:12:43.310544 13232 raft_consensus.cc:2454] T 236d2386bb2d4f378faeabce22238402 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:12:43.327538 19922 raft_consensus.cc:348] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.328588 19922 raft_consensus.cc:374] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.328783 19922 raft_consensus.cc:729] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.328935 12833 catalog_manager.cc:5484] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:43.329138 19922 consensus_queue.cc:260] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.329970 19922 ts_tablet_manager.cc:1398] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.019s user 0.000s sys 0.003s
I20240501 08:12:43.331209 19922 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:43.341183 19922 tablet_bootstrap.cc:654] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161: Neither blocks nor log segments found. Creating new log.
I20240501 08:12:43.350634 19943 raft_consensus.cc:482] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:43.350826 19943 raft_consensus.cc:504] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.351508 19943 leader_election.cc:288] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:43.352200 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66af91fdcce84b779cf6423562b582b0" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:43.352450 15754 raft_consensus.cc:2454] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 0.
I20240501 08:12:43.353060 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66af91fdcce84b779cf6423562b582b0" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:12:43.353848 15825 leader_election.cc:302] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
W20240501 08:12:43.354419 15825 leader_election.cc:341] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:43.354861 19943 raft_consensus.cc:2781] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:12:43.354954 19943 raft_consensus.cc:482] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:12:43.355018 19943 raft_consensus.cc:3037] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.357728 19943 raft_consensus.cc:504] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.358793 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66af91fdcce84b779cf6423562b582b0" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
W20240501 08:12:43.361121 15825 leader_election.cc:341] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Illegal state: must be running to vote when last-logged opid is not known
I20240501 08:12:43.366168 19922 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161: No bootstrap required, opened a new log
I20240501 08:12:43.366314 19922 ts_tablet_manager.cc:1367] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
I20240501 08:12:43.367278 19922 raft_consensus.cc:348] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.367467 19922 raft_consensus.cc:374] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:12:43.367583 19922 raft_consensus.cc:729] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:43.367827 19922 consensus_queue.cc:260] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.369693 19922 ts_tablet_manager.cc:1398] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:12:43.369868 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66af91fdcce84b779cf6423562b582b0" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:12:43.370093 15754 raft_consensus.cc:3037] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.374150 15754 raft_consensus.cc:2454] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:12:43.374595 15825 leader_election.cc:302] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters: 5dd722c770d34d6c9154a87917117161
I20240501 08:12:43.374953 19942 raft_consensus.cc:2781] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:12:43.375079 19942 raft_consensus.cc:686] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:12:43.375406 19942 consensus_queue.cc:237] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.379907 19943 leader_election.cc:288] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 1 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:43.380105 19663 raft_consensus.cc:482] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:43.380276 19663 raft_consensus.cc:504] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:43.380973 19663 leader_election.cc:288] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:43.381506 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6658ae0ccbcc4baf8853ab8778049030" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:12:43.382756 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6658ae0ccbcc4baf8853ab8778049030" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:12:43.382978 13232 raft_consensus.cc:2379] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5dd722c770d34d6c9154a87917117161 in current term 1: Already voted for candidate 4ecb724cb4764a53988fde00407db4c0 in this term.
I20240501 08:12:43.383395 15960 leader_election.cc:302] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:43.383914 19663 raft_consensus.cc:3037] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.387658 19663 raft_consensus.cc:2726] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:43.391160 12833 catalog_manager.cc:5484] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:43.450901 13364 consensus_queue.cc:237] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.453446 15891 raft_consensus.cc:1261] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.453956 19590 consensus_queue.cc:1035] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.454088 12971 raft_consensus.cc:1261] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:43.454553 19590 consensus_queue.cc:1035] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:43.456866 19590 raft_consensus.cc:2932] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:43.458053 19663 raft_consensus.cc:482] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:43.458207 19663 raft_consensus.cc:504] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:43.458906 19663 leader_election.cc:288] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:43.459249 15891 raft_consensus.cc:2932] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:43.461184 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66af91fdcce84b779cf6423562b582b0" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:12:43.461233 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66af91fdcce84b779cf6423562b582b0" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:12:43.461512 15754 raft_consensus.cc:2379] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5dd722c770d34d6c9154a87917117161 in current term 1: Already voted for candidate 4d516916eeac468db514eeb928f06190 in this term.
I20240501 08:12:43.461970 15959 leader_election.cc:302] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:43.462455 19404 raft_consensus.cc:3037] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.464516 19404 raft_consensus.cc:2726] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:43.465071 12971 raft_consensus.cc:2932] T d1af72684b304cab9a7d960084ea5926 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:43.469092 12833 catalog_manager.cc:5484] T d1af72684b304cab9a7d960084ea5926 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:43.476752 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet d1af72684b304cab9a7d960084ea5926 with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34524
I20240501 08:12:43.488343 19977 tablet_replica.cc:330] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:12:43.488564 19977 raft_consensus.cc:2229] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:43.488786 19977 raft_consensus.cc:2258] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:43.491737 19977 ts_tablet_manager.cc:1875] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:43.503885 19977 ts_tablet_manager.cc:1888] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:43.504137 19977 log.cc:1196] T d1af72684b304cab9a7d960084ea5926 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/d1af72684b304cab9a7d960084ea5926
I20240501 08:12:43.513113 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet d1af72684b304cab9a7d960084ea5926 (table rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]) successfully deleted
I20240501 08:12:43.514653 12830 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:46968:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240501 08:12:43.532618 15754 raft_consensus.cc:1261] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:43.533620 19942 consensus_queue.cc:1035] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:43.534098 16023 raft_consensus.cc:1261] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:43.535504 19943 consensus_queue.cc:1035] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:43.543431 12830 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 66af91fdcce84b779cf6423562b582b0 (table rebalancer_dynamic_table [id=b6b6af5609ce409ba8be11e257f107ef]). Expected version 1 got 0
I20240501 08:12:43.548738 12833 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet 66af91fdcce84b779cf6423562b582b0 (table rebalancer_dynamic_table [id=b6b6af5609ce409ba8be11e257f107ef]). Expected version 1 got 0
I20240501 08:12:43.550716 19983 mvcc.cc:205] Tried to move back new op lower bound from 7022801565813198848 to 7022801565188481024. Current Snapshot: MvccSnapshot[applied={T|T < 7022801565813198848}]
I20240501 08:12:43.554651 19988 tablet.cc:1722] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:43.557082 19988 tablet.cc:1620] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: MemRowSet was empty: no flush needed.
I20240501 08:12:43.558027 19758 raft_consensus.cc:482] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:12:43.558187 19989 tablet.cc:1722] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:43.558250 19758 raft_consensus.cc:504] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:43.558809 19989 tablet.cc:1620] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161: MemRowSet was empty: no flush needed.
I20240501 08:12:43.559226 19758 leader_election.cc:288] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:43.559944 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0cf43235be4fa9bf53b1067138bc72" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:12:43.562294 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da0cf43235be4fa9bf53b1067138bc72" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76d830e199eb461c8ca689c9460ee0c2" is_pre_election: true
I20240501 08:12:43.562559 12971 raft_consensus.cc:2379] T da0cf43235be4fa9bf53b1067138bc72 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 982f49fc87f149439cda0de2261b6bfd in current term 1: Already voted for candidate e3fefd4dca2d41678187047f06c10187 in this term.
I20240501 08:12:43.563216 15692 leader_election.cc:302] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187
I20240501 08:12:43.565544 19758 raft_consensus.cc:3037] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:12:43.569849 12830 catalog_manager.cc:5576] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897) does not have the latest schema for tablet 66af91fdcce84b779cf6423562b582b0 (table rebalancer_dynamic_table [id=b6b6af5609ce409ba8be11e257f107ef]). Expected version 1 got 0
I20240501 08:12:43.570129 19987 tablet.cc:1722] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:12:43.570523 19758 raft_consensus.cc:2726] T da0cf43235be4fa9bf53b1067138bc72 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 08:12:43.570840 19987 tablet.cc:1620] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190: MemRowSet was empty: no flush needed.
I20240501 08:12:43.573750 12833 catalog_manager.cc:5576] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997) does not have the latest schema for tablet 66af91fdcce84b779cf6423562b582b0 (table rebalancer_dynamic_table [id=b6b6af5609ce409ba8be11e257f107ef]). Expected version 1 got 0
I20240501 08:12:43.577847 19789 consensus_queue.cc:1035] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:43.587848 19972 consensus_queue.cc:1035] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:43.604950 12826 catalog_manager.cc:6029] T 00000000000000000000000000000000 P c48be6e775954c7093aaf62e68805ec0: rebalancer_dynamic_table [id=b6b6af5609ce409ba8be11e257f107ef] alter complete (version 1)
I20240501 08:12:43.677299 19941 consensus_queue.cc:1035] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:43.691648 19945 consensus_queue.cc:1035] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:43.708189 19404 consensus_queue.cc:1035] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:43.717168 19404 consensus_queue.cc:1035] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:12:43.928833 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:43.973837 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:43.973994 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:43.990689 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:43.997783 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:44.085345 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:44.113144 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:44.124711 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:44.316629 12971 consensus_queue.cc:237] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:44.318670 16023 raft_consensus.cc:1261] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:44.318838 13232 raft_consensus.cc:1261] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:44.319341 19818 consensus_queue.cc:1035] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:44.319609 19998 consensus_queue.cc:1035] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:44.322799 19818 raft_consensus.cc:2932] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:44.323541 13232 raft_consensus.cc:2932] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:44.323992 16023 raft_consensus.cc:2932] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:44.332904 12833 catalog_manager.cc:5484] T 7c4b83a76d7b44039ebb17245fcd8a3c P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:44.348955 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4b83a76d7b44039ebb17245fcd8a3c with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:44.351433 20087 tablet_replica.cc:330] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:44.351703 20087 raft_consensus.cc:2229] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:44.352028 20087 raft_consensus.cc:2258] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:44.352784 20087 ts_tablet_manager.cc:1875] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:44.358271 20087 ts_tablet_manager.cc:1888] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:44.358373 20087 log.cc:1196] T 7c4b83a76d7b44039ebb17245fcd8a3c P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/7c4b83a76d7b44039ebb17245fcd8a3c
I20240501 08:12:44.359031 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 7c4b83a76d7b44039ebb17245fcd8a3c (table rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]) successfully deleted
I20240501 08:12:44.666141 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:44.682139 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:44.684262 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:44.720130 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:44.793717 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:44.809145 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:44.809424 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:44.832491 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:45.130497 15891 consensus_queue.cc:237] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:45.133206 13232 raft_consensus.cc:1261] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:45.133867 19943 consensus_queue.cc:1035] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:45.134660 15756 raft_consensus.cc:1261] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:45.134675 15825 consensus_peers.cc:485] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c487b2c7fe24515a8970a07883c126f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:45.135334 20156 consensus_queue.cc:1035] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:45.138648 19943 raft_consensus.cc:2932] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:45.139267 15754 raft_consensus.cc:2932] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:45.148362 13232 raft_consensus.cc:2932] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:45.149061 12830 catalog_manager.cc:5484] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:45.412014 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:45.429157 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:45.477141 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:45.482787 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:45.530434 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:45.541699 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:45.558043 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:45.572041 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:45.692317 20227 ts_tablet_manager.cc:902] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:45.697166 20227 tablet_copy_client.cc:308] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:12:45.701305 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 8c487b2c7fe24515a8970a07883c126f from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:39283)
I20240501 08:12:45.701556 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 8c487b2c7fe24515a8970a07883c126f from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:39283: session id = e3fefd4dca2d41678187047f06c10187-8c487b2c7fe24515a8970a07883c126f
I20240501 08:12:45.705075 15913 tablet_copy_source_session.cc:205] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:45.709133 20227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c487b2c7fe24515a8970a07883c126f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:45.715005 20227 tablet_copy_client.cc:783] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:45.715325 20227 tablet_copy_client.cc:647] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:45.718293 20227 tablet_copy_client.cc:523] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:45.722460 20227 tablet_bootstrap.cc:492] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:45.731904 20227 tablet_bootstrap.cc:492] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:45.732542 20227 tablet_bootstrap.cc:492] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:12:45.732812 20227 ts_tablet_manager.cc:1367] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20240501 08:12:45.733858 20227 raft_consensus.cc:348] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:45.734117 20227 raft_consensus.cc:729] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:45.734400 20227 consensus_queue.cc:260] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:45.736914 20227 ts_tablet_manager.cc:1398] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:12:45.739377 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-8c487b2c7fe24515a8970a07883c126f received from {username='slave'} at 127.6.56.66:39283
I20240501 08:12:45.739581 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session e3fefd4dca2d41678187047f06c10187-8c487b2c7fe24515a8970a07883c126f on tablet 8c487b2c7fe24515a8970a07883c126f with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:12:45.976630 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:45.980312 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:45.988981 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:46.066848 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:46.078380 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:46.088757 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:46.097456 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:46.105765 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:46.257470 13104 raft_consensus.cc:1203] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:46.273150 12971 consensus_queue.cc:237] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:46.275238 15891 raft_consensus.cc:1261] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:46.275400 13104 raft_consensus.cc:1261] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:46.275781 20136 consensus_queue.cc:1035] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:46.276456 20136 consensus_queue.cc:1035] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:46.278975 19998 raft_consensus.cc:2932] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:46.279665 15891 raft_consensus.cc:2932] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:46.280213 13104 raft_consensus.cc:2932] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:46.286252 12832 catalog_manager.cc:5484] T d4ba66f68a134d09bc53d840ba8191d6 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:46.297502 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet d4ba66f68a134d09bc53d840ba8191d6 with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:46.311101 20314 tablet_replica.cc:330] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:46.311347 20314 raft_consensus.cc:2229] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:46.311758 20314 raft_consensus.cc:2258] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:46.312443 20314 ts_tablet_manager.cc:1875] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:46.317292 20314 ts_tablet_manager.cc:1888] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:46.317394 20314 log.cc:1196] T d4ba66f68a134d09bc53d840ba8191d6 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/d4ba66f68a134d09bc53d840ba8191d6
I20240501 08:12:46.318051 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet d4ba66f68a134d09bc53d840ba8191d6 (table rebalance_test_table_19 [id=ea3df3f0972a46c29ab991f3e578eb71]) successfully deleted
I20240501 08:12:46.758498 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:46.785831 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:46.820526 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:46.820953 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:46.841568 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:46.873626 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:46.873625 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:46.908627 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:47.088263 13104 consensus_queue.cc:237] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:47.097599 13232 raft_consensus.cc:1261] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:47.097795 15891 raft_consensus.cc:1261] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:47.098244 20269 consensus_queue.cc:1035] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.098539 20401 consensus_queue.cc:1035] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:47.101433 20402 raft_consensus.cc:2932] T d5179549829d451eb3a6e2083ef59b7f P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:47.104833 15891 raft_consensus.cc:2932] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:47.107342 13234 raft_consensus.cc:2932] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:47.113646 12830 catalog_manager.cc:5484] T d5179549829d451eb3a6e2083ef59b7f P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:47.121737 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet d5179549829d451eb3a6e2083ef59b7f with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:47.122457 20444 tablet_replica.cc:330] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:47.122678 20444 raft_consensus.cc:2229] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:47.122926 20444 raft_consensus.cc:2258] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:47.123646 20444 ts_tablet_manager.cc:1875] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:47.129781 20444 ts_tablet_manager.cc:1888] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:47.129897 20444 log.cc:1196] T d5179549829d451eb3a6e2083ef59b7f P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/d5179549829d451eb3a6e2083ef59b7f
I20240501 08:12:47.130651 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet d5179549829d451eb3a6e2083ef59b7f (table rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]) successfully deleted
I20240501 08:12:47.280560 13234 consensus_queue.cc:237] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:47.282970 13363 raft_consensus.cc:1261] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.283928 20508 consensus_queue.cc:1035] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:47.284876 13168 consensus_peers.cc:485] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6fefd76544642ddb787cc6febe8df82. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.285110 12971 raft_consensus.cc:1261] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.285928 20058 consensus_queue.cc:1035] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.288543 20508 raft_consensus.cc:2932] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.289127 12971 raft_consensus.cc:2932] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.292541 13363 raft_consensus.cc:2932] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.294720 12830 catalog_manager.cc:5484] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.343989 13234 consensus_queue.cc:237] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:47.346568 13105 raft_consensus.cc:1261] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.346561 13363 raft_consensus.cc:1261] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:47.346748 13170 consensus_peers.cc:485] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec6d272070b140d9b918d182613008f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.347013 19866 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.347234 20058 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.350304 20058 raft_consensus.cc:2932] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:47.350859 13363 raft_consensus.cc:2932] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:47.353961 13105 raft_consensus.cc:2932] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:47.359020 12833 catalog_manager.cc:5484] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:47.398941 16023 consensus_queue.cc:237] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:47.401741 16154 raft_consensus.cc:1261] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:47.402127 15959 consensus_peers.cc:485] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8953bcf510a2413cbceb4924f651b771. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.402516 20390 consensus_queue.cc:1035] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.403138 15754 raft_consensus.cc:1261] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.403618 20390 consensus_queue.cc:1035] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.406076 19477 raft_consensus.cc:2932] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:47.406731 15754 raft_consensus.cc:2932] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:47.410148 16154 raft_consensus.cc:2932] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:47.416271 12830 catalog_manager.cc:5484] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:47.453421 13105 consensus_queue.cc:237] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:47.455750 12971 raft_consensus.cc:1261] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.456115 13363 raft_consensus.cc:1261] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.456476 20269 consensus_queue.cc:1035] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:47.456743 13038 consensus_peers.cc:485] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13d9aa0c21e1405791e340cd79738256. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.457039 20401 consensus_queue.cc:1035] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.460716 20564 raft_consensus.cc:2932] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.461293 13364 raft_consensus.cc:2932] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.467777 12827 catalog_manager.cc:5484] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.467774 12971 raft_consensus.cc:2932] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.528668 13105 consensus_queue.cc:237] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
W20240501 08:12:47.541672 13041 consensus_peers.cc:485] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f723c6321424530bcd41b9ecdcbe0b3. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.542542 16023 raft_consensus.cc:1261] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.543080 15892 raft_consensus.cc:1261] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.543148 20564 consensus_queue.cc:1035] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.547209 20564 consensus_queue.cc:1035] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.561897 20269 raft_consensus.cc:2932] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:47.562759 15891 raft_consensus.cc:2932] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:47.567644 16024 raft_consensus.cc:2932] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:47.573343 12833 catalog_manager.cc:5484] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:47.610064 16024 consensus_queue.cc:237] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:47.612921 15891 raft_consensus.cc:1261] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.613026 12971 raft_consensus.cc:1261] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.613487 19477 consensus_queue.cc:1035] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:47.613790 15959 consensus_peers.cc:485] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc4ad07cecff43f0b867e4db444957e7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.614243 19477 consensus_queue.cc:1035] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.617466 19477 raft_consensus.cc:2932] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:47.618098 12971 raft_consensus.cc:2932] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:47.622339 15891 raft_consensus.cc:2932] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:47.624641 12827 catalog_manager.cc:5484] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:47.674301 15754 consensus_queue.cc:237] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:47.677009 13234 raft_consensus.cc:1261] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.677045 16024 raft_consensus.cc:1261] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.677780 19277 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:47.678232 15691 consensus_peers.cc:485] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6aa998326f3c4ced9afbee7e54bffa0e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.678601 19277 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.682025 19277 raft_consensus.cc:2932] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:47.682668 16024 raft_consensus.cc:2932] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:47.688858 12833 catalog_manager.cc:5484] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:47.692556 13234 raft_consensus.cc:2932] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:47.728487 13234 consensus_queue.cc:237] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:47.730945 12971 raft_consensus.cc:1261] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:12:47.731047 13168 consensus_peers.cc:485] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53872527ad264dd9b4b1627ff8aa46db. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.731503 13105 raft_consensus.cc:1261] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:47.731465 19866 consensus_queue.cc:1035] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.731976 20508 consensus_queue.cc:1035] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:47.735502 19866 raft_consensus.cc:2932] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.736554 13105 raft_consensus.cc:2932] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.742050 12974 raft_consensus.cc:2932] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.742291 12827 catalog_manager.cc:5484] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:47.780138 15891 consensus_queue.cc:237] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:47.782577 16024 raft_consensus.cc:1261] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:47.782964 15826 consensus_peers.cc:485] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37b0020baddc4f47b625edc2e1180e51. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.783149 20156 consensus_queue.cc:1035] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.783397 16157 raft_consensus.cc:1261] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.783895 20156 consensus_queue.cc:1035] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.786572 20159 raft_consensus.cc:2932] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:47.787184 16157 raft_consensus.cc:2932] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:47.796288 12830 catalog_manager.cc:5484] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:47.801906 16024 raft_consensus.cc:2932] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:47.844815 20677 ts_tablet_manager.cc:902] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:47.845043 16024 consensus_queue.cc:237] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:47.847791 12974 raft_consensus.cc:1261] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.848434 19477 consensus_queue.cc:1035] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:47.848729 15959 consensus_peers.cc:485] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d241e087e754485cb80080754d27aef7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:47.849493 15891 raft_consensus.cc:1261] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:47.850140 19477 consensus_queue.cc:1035] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:47.860770 20683 ts_tablet_manager.cc:902] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:12:47.865329 20683 tablet_copy_client.cc:308] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:12:47.865348 19477 raft_consensus.cc:2932] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:47.869006 13255 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet ec6d272070b140d9b918d182613008f4 from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:37937)
I20240501 08:12:47.869168 13255 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet ec6d272070b140d9b918d182613008f4 from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:37937: session id = 5dd722c770d34d6c9154a87917117161-ec6d272070b140d9b918d182613008f4
I20240501 08:12:47.871551 20677 tablet_copy_client.cc:308] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:12:47.871978 13255 tablet_copy_source_session.cc:205] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:47.872136 13254 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet e6fefd76544642ddb787cc6febe8df82 from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:54825)
I20240501 08:12:47.872344 13254 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet e6fefd76544642ddb787cc6febe8df82 from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:54825: session id = 6cc32f6e2d49477f9a23ce6e444039fb-e6fefd76544642ddb787cc6febe8df82
I20240501 08:12:47.874707 15891 raft_consensus.cc:2932] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:47.875181 13254 tablet_copy_source_session.cc:205] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:47.875293 20683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6d272070b140d9b918d182613008f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.885643 12830 catalog_manager.cc:5484] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:47.887696 20683 tablet_copy_client.cc:783] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:47.887956 20683 tablet_copy_client.cc:647] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:47.889549 20677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fefd76544642ddb787cc6febe8df82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.889588 20683 tablet_copy_client.cc:523] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:47.894966 20683 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:12:47.905640 20677 tablet_copy_client.cc:783] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:47.905931 20677 tablet_copy_client.cc:647] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:47.908239 20677 tablet_copy_client.cc:523] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:47.910638 12974 raft_consensus.cc:2932] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:47.913955 20677 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:47.961311 20693 ts_tablet_manager.cc:902] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:47.987120 20693 tablet_copy_client.cc:308] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:12:47.987975 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 8953bcf510a2413cbceb4924f651b771 from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:60749)
I20240501 08:12:47.988166 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 8953bcf510a2413cbceb4924f651b771 from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:60749: session id = e3fefd4dca2d41678187047f06c10187-8953bcf510a2413cbceb4924f651b771
I20240501 08:12:47.991837 16045 tablet_copy_source_session.cc:205] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:47.993649 20693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8953bcf510a2413cbceb4924f651b771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:47.994764 20701 ts_tablet_manager.cc:902] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:48.015166 20683 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.015659 20683 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:12:48.015944 20683 ts_tablet_manager.cc:1367] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.121s user 0.048s sys 0.007s
I20240501 08:12:48.016719 20683 raft_consensus.cc:348] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:48.023557 20683 raft_consensus.cc:729] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.024152 20683 consensus_queue.cc:260] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:48.025332 20683 ts_tablet_manager.cc:1398] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:12:48.026485 13254 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-ec6d272070b140d9b918d182613008f4 received from {username='slave'} at 127.6.56.71:37937
I20240501 08:12:48.026592 13254 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 5dd722c770d34d6c9154a87917117161-ec6d272070b140d9b918d182613008f4 on tablet ec6d272070b140d9b918d182613008f4 with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:12:48.039448 20693 tablet_copy_client.cc:783] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.039832 20693 tablet_copy_client.cc:647] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.044000 20693 tablet_copy_client.cc:523] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.052841 20693 tablet_bootstrap.cc:492] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:48.096494 20677 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.097406 20677 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:12:48.097715 20677 ts_tablet_manager.cc:1367] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.184s user 0.065s sys 0.000s
I20240501 08:12:48.098279 20693 tablet_bootstrap.cc:492] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.098716 20693 tablet_bootstrap.cc:492] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:12:48.098819 20677 raft_consensus.cc:348] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:48.099097 20693 ts_tablet_manager.cc:1367] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.000s
I20240501 08:12:48.099104 20677 raft_consensus.cc:729] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.099472 20677 consensus_queue.cc:260] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:48.100329 20677 ts_tablet_manager.cc:1398] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:48.102380 13254 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-e6fefd76544642ddb787cc6febe8df82 received from {username='slave'} at 127.6.56.72:54825
I20240501 08:12:48.102511 13254 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-e6fefd76544642ddb787cc6febe8df82 on tablet e6fefd76544642ddb787cc6febe8df82 with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:48.120180 20693 raft_consensus.cc:348] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:48.120558 20693 raft_consensus.cc:729] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.121582 20693 consensus_queue.cc:260] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:48.123265 20693 ts_tablet_manager.cc:1398] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240501 08:12:48.127600 20712 ts_tablet_manager.cc:902] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:48.128814 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-8953bcf510a2413cbceb4924f651b771 received from {username='slave'} at 127.6.56.66:60749
I20240501 08:12:48.128958 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session e3fefd4dca2d41678187047f06c10187-8953bcf510a2413cbceb4924f651b771 on tablet 8953bcf510a2413cbceb4924f651b771 with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:12:48.135370 20711 ts_tablet_manager.cc:902] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:48.136876 20701 tablet_copy_client.cc:308] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:12:48.137508 13125 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet 13d9aa0c21e1405791e340cd79738256 from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:42629)
I20240501 08:12:48.137668 13125 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet 13d9aa0c21e1405791e340cd79738256 from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:42629: session id = 6cc32f6e2d49477f9a23ce6e444039fb-13d9aa0c21e1405791e340cd79738256
I20240501 08:12:48.141067 13125 tablet_copy_source_session.cc:205] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.145601 20701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13d9aa0c21e1405791e340cd79738256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.154083 20701 tablet_copy_client.cc:783] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.154484 20701 tablet_copy_client.cc:647] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.156612 20701 tablet_copy_client.cc:523] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.160218 20701 tablet_bootstrap.cc:492] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:48.189365 20677 ts_tablet_manager.cc:902] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:12:48.199647 12910 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 52996.1us
I20240501 08:12:48.254393 20712 tablet_copy_client.cc:308] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:12:48.257799 20693 ts_tablet_manager.cc:902] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:48.258262 13125 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet 0f723c6321424530bcd41b9ecdcbe0b3 from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:33769)
I20240501 08:12:48.258445 13125 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet 0f723c6321424530bcd41b9ecdcbe0b3 from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:33769: session id = 6ed448d04e4646658c0de37d829ff15b-0f723c6321424530bcd41b9ecdcbe0b3
I20240501 08:12:48.259425 20693 tablet_copy_client.cc:308] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:12:48.260469 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet 6aa998326f3c4ced9afbee7e54bffa0e from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:33955)
I20240501 08:12:48.260684 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet 6aa998326f3c4ced9afbee7e54bffa0e from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:33955: session id = e3fefd4dca2d41678187047f06c10187-6aa998326f3c4ced9afbee7e54bffa0e
I20240501 08:12:48.266283 13125 tablet_copy_source_session.cc:205] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.270289 15776 tablet_copy_source_session.cc:205] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.278093 20712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f723c6321424530bcd41b9ecdcbe0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.283099 16024 raft_consensus.cc:1203] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:48.293262 20693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa998326f3c4ced9afbee7e54bffa0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.298604 20677 tablet_copy_client.cc:308] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:12:48.299208 13254 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet 53872527ad264dd9b4b1627ff8aa46db from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:54825)
I20240501 08:12:48.299362 13254 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet 53872527ad264dd9b4b1627ff8aa46db from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:54825: session id = 6cc32f6e2d49477f9a23ce6e444039fb-53872527ad264dd9b4b1627ff8aa46db
I20240501 08:12:48.302242 13254 tablet_copy_source_session.cc:205] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.305435 20677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53872527ad264dd9b4b1627ff8aa46db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.307602 20712 tablet_copy_client.cc:783] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.307862 20712 tablet_copy_client.cc:647] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.314193 20677 tablet_copy_client.cc:783] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.314431 20677 tablet_copy_client.cc:647] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.316620 20677 tablet_copy_client.cc:523] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.320407 20677 tablet_bootstrap.cc:492] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:12:48.324072 20712 tablet_copy_client.cc:523] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.327777 20712 tablet_bootstrap.cc:492] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:48.337484 20701 tablet_bootstrap.cc:492] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.339399 20701 tablet_bootstrap.cc:492] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:12:48.340072 20701 ts_tablet_manager.cc:1367] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.180s user 0.050s sys 0.014s
I20240501 08:12:48.346558 20701 raft_consensus.cc:348] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:48.346856 20701 raft_consensus.cc:729] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.347139 20701 consensus_queue.cc:260] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:48.347724 20701 ts_tablet_manager.cc:1398] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 08:12:48.350360 20712 tablet_bootstrap.cc:492] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.350788 20712 tablet_bootstrap.cc:492] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:12:48.351027 20712 ts_tablet_manager.cc:1367] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.001s
I20240501 08:12:48.351751 20712 raft_consensus.cc:348] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:48.351965 20712 raft_consensus.cc:729] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.352326 20712 consensus_queue.cc:260] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:48.356101 20712 ts_tablet_manager.cc:1398] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:12:48.357020 13125 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-13d9aa0c21e1405791e340cd79738256 received from {username='slave'} at 127.6.56.72:42629
I20240501 08:12:48.357137 13125 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-13d9aa0c21e1405791e340cd79738256 on tablet 13d9aa0c21e1405791e340cd79738256 with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:48.357842 13125 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-0f723c6321424530bcd41b9ecdcbe0b3 received from {username='slave'} at 127.6.56.67:33769
I20240501 08:12:48.357954 13125 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-0f723c6321424530bcd41b9ecdcbe0b3 on tablet 0f723c6321424530bcd41b9ecdcbe0b3 with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:48.381942 20693 tablet_copy_client.cc:783] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.382223 20693 tablet_copy_client.cc:647] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.387260 20693 tablet_copy_client.cc:523] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.394472 20677 tablet_bootstrap.cc:492] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.394753 20711 tablet_copy_client.cc:308] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:12:48.394989 20712 ts_tablet_manager.cc:902] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:48.396348 15891 consensus_queue.cc:237] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:48.397554 20693 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:48.397938 13105 raft_consensus.cc:1203] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:48.398610 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet fc4ad07cecff43f0b867e4db444957e7 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:50471)
I20240501 08:12:48.398761 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet fc4ad07cecff43f0b867e4db444957e7 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:50471: session id = 4ecb724cb4764a53988fde00407db4c0-fc4ad07cecff43f0b867e4db444957e7
I20240501 08:12:48.401508 16045 tablet_copy_source_session.cc:205] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.403560 20711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4ad07cecff43f0b867e4db444957e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.395063 20677 tablet_bootstrap.cc:492] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:12:48.405509 20677 ts_tablet_manager.cc:1367] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.085s user 0.057s sys 0.000s
I20240501 08:12:48.406457 20677 raft_consensus.cc:348] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:48.406836 20677 raft_consensus.cc:729] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.407163 20677 consensus_queue.cc:260] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:48.412066 20677 ts_tablet_manager.cc:1398] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:12:48.412655 20711 tablet_copy_client.cc:783] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.412981 20711 tablet_copy_client.cc:647] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.414160 13254 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-53872527ad264dd9b4b1627ff8aa46db received from {username='slave'} at 127.6.56.72:54825
I20240501 08:12:48.414283 13254 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-53872527ad264dd9b4b1627ff8aa46db on tablet 53872527ad264dd9b4b1627ff8aa46db with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:48.420135 20711 tablet_copy_client.cc:523] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.424456 20711 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:48.424856 20693 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.425233 20693 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:12:48.425485 20693 ts_tablet_manager.cc:1367] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.011s
I20240501 08:12:48.426173 20693 raft_consensus.cc:348] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:48.426403 20693 raft_consensus.cc:729] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.426960 20693 consensus_queue.cc:260] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:48.427491 20693 ts_tablet_manager.cc:1398] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:48.428223 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-6aa998326f3c4ced9afbee7e54bffa0e received from {username='slave'} at 127.6.56.66:33955
I20240501 08:12:48.428460 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session e3fefd4dca2d41678187047f06c10187-6aa998326f3c4ced9afbee7e54bffa0e on tablet 6aa998326f3c4ced9afbee7e54bffa0e with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:12:48.432685 15756 raft_consensus.cc:1261] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.433216 13363 raft_consensus.cc:1261] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.433781 20660 consensus_queue.cc:1035] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.435778 20660 consensus_queue.cc:1035] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.438973 16157 raft_consensus.cc:1203] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 08:12:48.445482 15826 consensus_peers.cc:485] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8a350eade1c41e4b4c29e487374552f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.447643 20711 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.448177 20711 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:12:48.448527 20711 ts_tablet_manager.cc:1367] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.005s
I20240501 08:12:48.449499 20711 raft_consensus.cc:348] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:48.449714 20712 tablet_copy_client.cc:308] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:12:48.449849 20711 raft_consensus.cc:729] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.450085 20711 consensus_queue.cc:260] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:48.450425 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 37b0020baddc4f47b625edc2e1180e51 from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:42849)
I20240501 08:12:48.450600 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 37b0020baddc4f47b625edc2e1180e51 from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:42849: session id = 6ed448d04e4646658c0de37d829ff15b-37b0020baddc4f47b625edc2e1180e51
I20240501 08:12:48.450713 20711 ts_tablet_manager.cc:1398] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:48.453725 15913 tablet_copy_source_session.cc:205] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.456370 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-fc4ad07cecff43f0b867e4db444957e7 received from {username='slave'} at 127.6.56.68:50471
I20240501 08:12:48.456507 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-fc4ad07cecff43f0b867e4db444957e7 on tablet fc4ad07cecff43f0b867e4db444957e7 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:12:48.458905 20712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b0020baddc4f47b625edc2e1180e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.466334 20711 ts_tablet_manager.cc:902] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:48.468683 20712 tablet_copy_client.cc:783] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.469009 20712 tablet_copy_client.cc:647] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.473213 20711 tablet_copy_client.cc:308] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:12:48.473927 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet d241e087e754485cb80080754d27aef7 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:50471)
I20240501 08:12:48.474117 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet d241e087e754485cb80080754d27aef7 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:50471: session id = 4ecb724cb4764a53988fde00407db4c0-d241e087e754485cb80080754d27aef7
I20240501 08:12:48.477979 16045 tablet_copy_source_session.cc:205] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:48.478574 20712 tablet_copy_client.cc:523] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.479429 20711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d241e087e754485cb80080754d27aef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:48.482995 20712 tablet_bootstrap.cc:492] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:48.486577 20711 tablet_copy_client.cc:783] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:48.486864 20711 tablet_copy_client.cc:647] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:48.488502 20711 tablet_copy_client.cc:523] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:48.495724 20711 tablet_bootstrap.cc:492] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:12:48.507952 16154 raft_consensus.cc:1203] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:48.537974 20660 raft_consensus.cc:2932] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:48.541237 15754 raft_consensus.cc:2932] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:48.548574 20711 tablet_bootstrap.cc:492] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.549610 20711 tablet_bootstrap.cc:492] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:12:48.550323 13364 raft_consensus.cc:2932] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:48.551110 20711 ts_tablet_manager.cc:1367] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.056s user 0.014s sys 0.019s
I20240501 08:12:48.552645 20711 raft_consensus.cc:348] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:48.554338 20711 raft_consensus.cc:729] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.554759 20711 consensus_queue.cc:260] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:48.558311 20711 ts_tablet_manager.cc:1398] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:12:48.559041 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-d241e087e754485cb80080754d27aef7 received from {username='slave'} at 127.6.56.68:50471
I20240501 08:12:48.559181 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-d241e087e754485cb80080754d27aef7 on tablet d241e087e754485cb80080754d27aef7 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:12:48.560032 12833 catalog_manager.cc:5484] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:48.567070 20712 tablet_bootstrap.cc:492] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:48.567597 20712 tablet_bootstrap.cc:492] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:12:48.567924 20712 ts_tablet_manager.cc:1367] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.085s user 0.032s sys 0.039s
I20240501 08:12:48.568946 20712 raft_consensus.cc:348] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:48.569205 20712 raft_consensus.cc:729] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:48.569527 20712 consensus_queue.cc:260] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:48.570171 20712 ts_tablet_manager.cc:1398] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240501 08:12:48.576586 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-37b0020baddc4f47b625edc2e1180e51 received from {username='slave'} at 127.6.56.67:42849
I20240501 08:12:48.576740 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-37b0020baddc4f47b625edc2e1180e51 on tablet 37b0020baddc4f47b625edc2e1180e51 with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:48.598083 13364 raft_consensus.cc:1203] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:48.647045 13232 raft_consensus.cc:1203] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:48.652354 13364 consensus_queue.cc:237] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:48.655263 15756 raft_consensus.cc:1261] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.655418 16157 raft_consensus.cc:1261] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.656103 19972 consensus_queue.cc:1035] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:48.656431 13300 consensus_peers.cc:485] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d16de310e7148768967be392b7af2ee. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.657397 19789 consensus_queue.cc:1035] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.660109 19789 raft_consensus.cc:2932] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:48.660930 15756 raft_consensus.cc:2932] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:48.666524 16157 raft_consensus.cc:2932] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:48.670194 12830 catalog_manager.cc:5484] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:48.728075 15891 consensus_queue.cc:237] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:48.730571 16024 raft_consensus.cc:1261] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.730612 12974 raft_consensus.cc:1261] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.731182 20660 consensus_queue.cc:1035] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:48.731637 15826 consensus_peers.cc:485] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa10a6f53f7f48cf9410c80e1056439d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.731644 20159 consensus_queue.cc:1035] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.735697 20159 raft_consensus.cc:2932] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:48.740350 12974 raft_consensus.cc:2932] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:48.739717 16024 raft_consensus.cc:2932] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:48.746337 12831 catalog_manager.cc:5484] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:48.782888 16157 raft_consensus.cc:1203] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:48.786327 12974 consensus_queue.cc:237] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:48.788648 13232 raft_consensus.cc:1261] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:48.789146 13364 raft_consensus.cc:1261] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:12:48.789160 20721 consensus_queue.cc:1035] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:12:48.791062 12907 consensus_peers.cc:485] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8156a46917c40e2b836a603ff70717a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.791136 20721 consensus_queue.cc:1035] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:12:48.795269 20726 raft_consensus.cc:2932] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:48.795758 13363 raft_consensus.cc:2932] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:48.801366 13232 raft_consensus.cc:2932] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:48.802856 12831 catalog_manager.cc:5484] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:48.842048 16024 consensus_queue.cc:237] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:48.844377 16157 raft_consensus.cc:1261] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:48.844789 15960 consensus_peers.cc:485] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03895995b2e5483998d442855006b24c. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.844966 20799 consensus_queue.cc:1035] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.845219 15756 raft_consensus.cc:1261] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.845779 19477 consensus_queue.cc:1035] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.848181 20799 raft_consensus.cc:2932] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:48.848886 15756 raft_consensus.cc:2932] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:48.855438 12827 catalog_manager.cc:5484] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:48.855808 16157 raft_consensus.cc:2932] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:48.864377 13104 raft_consensus.cc:1203] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:48.888170 13232 raft_consensus.cc:1203] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:48.934278 16024 consensus_queue.cc:237] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:48.937981 13104 raft_consensus.cc:1261] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.938623 19477 consensus_queue.cc:1035] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:12:48.938922 15960 consensus_peers.cc:485] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a71b0688ca04973a46b3cbc2ce680d2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:48.939965 16157 raft_consensus.cc:1261] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:48.940507 19477 consensus_queue.cc:1035] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:48.942970 13363 raft_consensus.cc:1203] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:48.947044 20547 raft_consensus.cc:2932] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:48.951792 16157 raft_consensus.cc:2932] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:48.954706 13104 raft_consensus.cc:2932] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:48.955776 12827 catalog_manager.cc:5484] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:49.005729 15756 consensus_queue.cc:237] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:49.008246 12974 raft_consensus.cc:1261] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:12:49.008337 16024 raft_consensus.cc:1261] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:12:49.008487 15691 consensus_peers.cc:485] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 321bf6596ff24ec7af79e0510eadedce. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:12:49.008960 20877 consensus_queue.cc:1035] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:49.009351 20733 consensus_queue.cc:1035] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:12:49.012718 20877 raft_consensus.cc:2932] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:49.013331 12974 raft_consensus.cc:2932] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:49.017828 16024 raft_consensus.cc:2932] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:49.018903 12827 catalog_manager.cc:5484] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:49.054095 20882 ts_tablet_manager.cc:902] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:49.104445 20882 tablet_copy_client.cc:308] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:12:49.105944 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet b8a350eade1c41e4b4c29e487374552f from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:37251)
I20240501 08:12:49.106200 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet b8a350eade1c41e4b4c29e487374552f from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:37251: session id = 76d830e199eb461c8ca689c9460ee0c2-b8a350eade1c41e4b4c29e487374552f
I20240501 08:12:49.109928 15913 tablet_copy_source_session.cc:205] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:49.111397 20882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8a350eade1c41e4b4c29e487374552f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:49.120041 20882 tablet_copy_client.cc:783] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:49.120309 20882 tablet_copy_client.cc:647] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:49.122646 20896 ts_tablet_manager.cc:902] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:12:49.122756 20882 tablet_copy_client.cc:523] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:49.126638 20882 tablet_bootstrap.cc:492] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:49.143239 20882 tablet_bootstrap.cc:492] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:49.143622 20882 tablet_bootstrap.cc:492] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:12:49.143955 20882 ts_tablet_manager.cc:1367] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.004s
I20240501 08:12:49.144711 20882 raft_consensus.cc:348] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:49.144942 20882 raft_consensus.cc:729] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:49.145184 20882 consensus_queue.cc:260] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:49.145721 20882 ts_tablet_manager.cc:1398] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:12:49.146360 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-b8a350eade1c41e4b4c29e487374552f received from {username='slave'} at 127.6.56.65:37251
I20240501 08:12:49.146548 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-b8a350eade1c41e4b4c29e487374552f on tablet b8a350eade1c41e4b4c29e487374552f with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:49.156590 20896 tablet_copy_client.cc:308] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:12:49.157361 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet fa10a6f53f7f48cf9410c80e1056439d from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:42849)
I20240501 08:12:49.157554 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet fa10a6f53f7f48cf9410c80e1056439d from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:42849: session id = 6ed448d04e4646658c0de37d829ff15b-fa10a6f53f7f48cf9410c80e1056439d
I20240501 08:12:49.159888 15913 tablet_copy_source_session.cc:205] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:49.162691 20896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa10a6f53f7f48cf9410c80e1056439d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:49.170714 20896 tablet_copy_client.cc:783] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:49.170970 20896 tablet_copy_client.cc:647] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:49.172530 20896 tablet_copy_client.cc:523] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:49.175699 20896 tablet_bootstrap.cc:492] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:49.181964 20896 tablet_bootstrap.cc:492] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:49.182365 20896 tablet_bootstrap.cc:492] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:12:49.182641 20896 ts_tablet_manager.cc:1367] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20240501 08:12:49.183435 20896 raft_consensus.cc:348] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:49.183665 20896 raft_consensus.cc:729] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:49.183915 20896 consensus_queue.cc:260] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:49.184564 20896 ts_tablet_manager.cc:1398] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:12:49.185595 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-fa10a6f53f7f48cf9410c80e1056439d received from {username='slave'} at 127.6.56.67:42849
I20240501 08:12:49.185726 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-fa10a6f53f7f48cf9410c80e1056439d on tablet fa10a6f53f7f48cf9410c80e1056439d with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:49.290756 20914 ts_tablet_manager.cc:902] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:12:49.324965 20914 tablet_copy_client.cc:308] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:12:49.325773 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 3d16de310e7148768967be392b7af2ee from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:41029)
I20240501 08:12:49.325923 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 3d16de310e7148768967be392b7af2ee from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:41029: session id = e3fefd4dca2d41678187047f06c10187-3d16de310e7148768967be392b7af2ee
I20240501 08:12:49.328184 13385 tablet_copy_source_session.cc:205] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:49.330474 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d16de310e7148768967be392b7af2ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:49.346480 20924 ts_tablet_manager.cc:902] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:12:49.349028 20914 tablet_copy_client.cc:783] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:49.349287 20914 tablet_copy_client.cc:647] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:49.360028 20914 tablet_copy_client.cc:523] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:49.361308 20924 tablet_copy_client.cc:308] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:12:49.362114 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet a8156a46917c40e2b836a603ff70717a from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:38689)
I20240501 08:12:49.362323 20896 ts_tablet_manager.cc:902] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:49.362306 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet a8156a46917c40e2b836a603ff70717a from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:38689: session id = 4d516916eeac468db514eeb928f06190-a8156a46917c40e2b836a603ff70717a
I20240501 08:12:49.364001 20914 tablet_bootstrap.cc:492] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:49.365859 12994 tablet_copy_source_session.cc:205] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:49.370574 20924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8156a46917c40e2b836a603ff70717a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:49.382421 20924 tablet_copy_client.cc:783] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:49.382881 20924 tablet_copy_client.cc:647] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:49.385473 20924 tablet_copy_client.cc:523] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:49.390084 20924 tablet_bootstrap.cc:492] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:12:49.394644 20914 tablet_bootstrap.cc:492] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:49.397553 20914 tablet_bootstrap.cc:492] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:12:49.397879 20914 ts_tablet_manager.cc:1367] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.013s
I20240501 08:12:49.398747 20914 raft_consensus.cc:348] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:49.398994 20914 raft_consensus.cc:729] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:49.399286 20914 consensus_queue.cc:260] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:49.400043 20914 ts_tablet_manager.cc:1398] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:12:49.400852 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-3d16de310e7148768967be392b7af2ee received from {username='slave'} at 127.6.56.66:41029
I20240501 08:12:49.400986 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session e3fefd4dca2d41678187047f06c10187-3d16de310e7148768967be392b7af2ee on tablet 3d16de310e7148768967be392b7af2ee with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:12:49.406584 20896 tablet_copy_client.cc:308] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:12:49.408207 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 7a71b0688ca04973a46b3cbc2ce680d2 from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:56345)
I20240501 08:12:49.408422 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 7a71b0688ca04973a46b3cbc2ce680d2 from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:56345: session id = 6ed448d04e4646658c0de37d829ff15b-7a71b0688ca04973a46b3cbc2ce680d2
I20240501 08:12:49.412178 16045 tablet_copy_source_session.cc:205] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:49.423092 20882 ts_tablet_manager.cc:902] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:12:49.427712 20896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a71b0688ca04973a46b3cbc2ce680d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:49.450265 20914 ts_tablet_manager.cc:902] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:12:49.478621 20882 tablet_copy_client.cc:308] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:12:49.479694 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 03895995b2e5483998d442855006b24c from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:52501)
I20240501 08:12:49.480020 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 03895995b2e5483998d442855006b24c from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:52501: session id = 76d830e199eb461c8ca689c9460ee0c2-03895995b2e5483998d442855006b24c
I20240501 08:12:49.484227 20924 tablet_bootstrap.cc:492] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:49.488034 20924 tablet_bootstrap.cc:492] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:12:49.488409 20924 ts_tablet_manager.cc:1367] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.098s user 0.065s sys 0.006s
I20240501 08:12:49.489375 20924 raft_consensus.cc:348] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:49.489717 20924 raft_consensus.cc:729] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:12:49.490001 20924 consensus_queue.cc:260] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:49.490751 20924 ts_tablet_manager.cc:1398] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:12:49.491730 16045 tablet_copy_source_session.cc:205] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:49.496821 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-a8156a46917c40e2b836a603ff70717a received from {username='slave'} at 127.6.56.70:38689
I20240501 08:12:49.496987 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 4d516916eeac468db514eeb928f06190-a8156a46917c40e2b836a603ff70717a on tablet a8156a46917c40e2b836a603ff70717a with peer 4d516916eeac468db514eeb928f06190
I20240501 08:12:49.497342 20882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03895995b2e5483998d442855006b24c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:49.508648 12973 raft_consensus.cc:1203] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.517396 13232 raft_consensus.cc:1203] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.519285 20882 tablet_copy_client.cc:783] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:49.519543 20882 tablet_copy_client.cc:647] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:49.521733 20882 tablet_copy_client.cc:523] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:49.526391 20882 tablet_bootstrap.cc:492] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:12:49.535451 20896 tablet_copy_client.cc:783] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:49.535914 20896 tablet_copy_client.cc:647] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:49.546219 20882 tablet_bootstrap.cc:492] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:49.546619 20882 tablet_bootstrap.cc:492] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:12:49.546869 20882 ts_tablet_manager.cc:1367] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20240501 08:12:49.547586 20882 raft_consensus.cc:348] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:49.547801 20882 raft_consensus.cc:729] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:12:49.548089 20882 consensus_queue.cc:260] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:49.551580 20896 tablet_copy_client.cc:523] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:49.552389 20882 ts_tablet_manager.cc:1398] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240501 08:12:49.554286 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-03895995b2e5483998d442855006b24c received from {username='slave'} at 127.6.56.65:52501
I20240501 08:12:49.554420 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-03895995b2e5483998d442855006b24c on tablet 03895995b2e5483998d442855006b24c with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:12:49.557816 20896 tablet_bootstrap.cc:492] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:12:49.577863 20914 tablet_copy_client.cc:308] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:12:49.578706 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet 321bf6596ff24ec7af79e0510eadedce from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:33955)
I20240501 08:12:49.578882 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet 321bf6596ff24ec7af79e0510eadedce from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:33955: session id = e3fefd4dca2d41678187047f06c10187-321bf6596ff24ec7af79e0510eadedce
I20240501 08:12:49.581708 15776 tablet_copy_source_session.cc:205] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:12:49.583101 20914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321bf6596ff24ec7af79e0510eadedce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:12:49.615861 20896 tablet_bootstrap.cc:492] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:49.616498 20896 tablet_bootstrap.cc:492] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:12:49.616787 20896 ts_tablet_manager.cc:1367] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.005s
I20240501 08:12:49.618245 20914 tablet_copy_client.cc:783] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:12:49.618522 20914 tablet_copy_client.cc:647] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:12:49.617630 20896 raft_consensus.cc:348] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:49.619148 20896 raft_consensus.cc:729] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:12:49.619426 20896 consensus_queue.cc:260] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:49.621654 20896 ts_tablet_manager.cc:1398] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:12:49.623400 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-7a71b0688ca04973a46b3cbc2ce680d2 received from {username='slave'} at 127.6.56.67:56345
I20240501 08:12:49.623533 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-7a71b0688ca04973a46b3cbc2ce680d2 on tablet 7a71b0688ca04973a46b3cbc2ce680d2 with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:12:49.625797 20914 tablet_copy_client.cc:523] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:12:49.629889 20914 tablet_bootstrap.cc:492] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:12:49.647015 20914 tablet_bootstrap.cc:492] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:12:49.649354 20914 tablet_bootstrap.cc:492] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:12:49.650197 20914 ts_tablet_manager.cc:1367] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240501 08:12:49.653705 20914 raft_consensus.cc:348] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:49.654404 20914 raft_consensus.cc:729] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:12:49.654959 20914 consensus_queue.cc:260] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:49.656123 20914 ts_tablet_manager.cc:1398] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 08:12:49.658011 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-321bf6596ff24ec7af79e0510eadedce received from {username='slave'} at 127.6.56.66:33955
I20240501 08:12:49.658157 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session e3fefd4dca2d41678187047f06c10187-321bf6596ff24ec7af79e0510eadedce on tablet 321bf6596ff24ec7af79e0510eadedce with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:12:49.707111 13104 raft_consensus.cc:1203] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.757408 13232 raft_consensus.cc:1203] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.763407 15891 raft_consensus.cc:1203] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:12:49.812213 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:49.816108 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:49.839221 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:49.848260 12973 raft_consensus.cc:1203] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:49.848325 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:49.865660 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:49.881878 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:49.933177 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:49.986649 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:50.064324 13105 raft_consensus.cc:1203] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:12:50.407013 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:50.423381 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:50.450538 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:50.466126 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:50.475068 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 08:12:50.532215 15826 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 63945.6us
I20240501 08:12:50.601629 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:50.604267 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 08:12:50.640058 13168 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 92603.6us
I20240501 08:12:50.675722 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:50.805740 16024 consensus_queue.cc:237] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:50.807796 13232 raft_consensus.cc:1261] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:50.808025 13104 raft_consensus.cc:1261] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:50.808665 20547 consensus_queue.cc:1035] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:50.809551 20799 consensus_queue.cc:1035] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240501 08:12:50.815330 20547 raft_consensus.cc:2932] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:50.815855 13232 raft_consensus.cc:2932] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:50.816563 13104 raft_consensus.cc:2932] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:50.822726 12833 catalog_manager.cc:5484] T 7a71b0688ca04973a46b3cbc2ce680d2 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:50.831633 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a71b0688ca04973a46b3cbc2ce680d2 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:50.834306 21089 tablet_replica.cc:330] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:50.834542 21089 raft_consensus.cc:2229] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:50.834822 21089 raft_consensus.cc:2258] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:50.835505 21089 ts_tablet_manager.cc:1875] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:50.843871 21089 ts_tablet_manager.cc:1888] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:50.844077 21089 log.cc:1196] T 7a71b0688ca04973a46b3cbc2ce680d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/7a71b0688ca04973a46b3cbc2ce680d2
I20240501 08:12:50.844856 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 7a71b0688ca04973a46b3cbc2ce680d2 (table rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]) successfully deleted
I20240501 08:12:51.402150 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:51.402781 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:51.417586 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:51.461015 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:51.473302 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:51.512943 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:12:51.560645 13040 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.69:42897, user_credentials={real_user=slave}} blocked reactor thread for 113842us
W20240501 08:12:51.564184 13298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 113725us
I20240501 08:12:51.619330 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:51.637521 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:51.751883 16024 consensus_queue.cc:237] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:51.754138 16154 raft_consensus.cc:1261] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.754165 12973 raft_consensus.cc:1261] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:51.754776 21083 consensus_queue.cc:1035] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.755075 20547 consensus_queue.cc:1035] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:51.771713 20799 raft_consensus.cc:2932] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:51.772413 12973 raft_consensus.cc:2932] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:51.777271 16154 raft_consensus.cc:2932] T 03895995b2e5483998d442855006b24c P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:51.779572 12830 catalog_manager.cc:5484] T 03895995b2e5483998d442855006b24c P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:51.784333 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 03895995b2e5483998d442855006b24c with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:51.793296 21178 tablet_replica.cc:330] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:51.793552 21178 raft_consensus.cc:2229] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:51.793849 21178 raft_consensus.cc:2258] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:51.794526 21178 ts_tablet_manager.cc:1875] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:51.800354 21178 ts_tablet_manager.cc:1888] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:51.800518 21178 log.cc:1196] T 03895995b2e5483998d442855006b24c P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/03895995b2e5483998d442855006b24c
I20240501 08:12:51.801316 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 03895995b2e5483998d442855006b24c (table rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]) successfully deleted
I20240501 08:12:52.169962 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:52.174149 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:52.206226 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:52.218283 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:52.219648 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:52.256568 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:52.260219 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:52.318576 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:52.442680 12971 consensus_queue.cc:237] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:12:52.444754 13232 raft_consensus.cc:1261] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:52.444801 15891 raft_consensus.cc:1261] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:52.445313 21249 consensus_queue.cc:1035] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:52.445601 21137 consensus_queue.cc:1035] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:52.448590 21137 raft_consensus.cc:2932] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:52.457188 12831 catalog_manager.cc:5484] T a8156a46917c40e2b836a603ff70717a P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:52.458562 13232 raft_consensus.cc:2932] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:52.471779 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet a8156a46917c40e2b836a603ff70717a with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:12:52.472537 21273 tablet_replica.cc:330] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:12:52.472914 21273 raft_consensus.cc:2229] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:52.473268 21273 raft_consensus.cc:2258] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:52.473279 15891 raft_consensus.cc:2932] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:12:52.474155 21273 ts_tablet_manager.cc:1875] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:52.484781 21273 ts_tablet_manager.cc:1888] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:52.484969 21273 log.cc:1196] T a8156a46917c40e2b836a603ff70717a P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/a8156a46917c40e2b836a603ff70717a
I20240501 08:12:52.486037 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet a8156a46917c40e2b836a603ff70717a (table rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]) successfully deleted
I20240501 08:12:52.763211 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:52.778769 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:52.788551 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:52.808337 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:52.810783 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:52.852977 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:52.896654 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:52.947302 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:53.071177 15891 consensus_queue.cc:237] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:12:53.073463 13363 raft_consensus.cc:1261] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:53.073444 12973 raft_consensus.cc:1261] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:53.074000 20660 consensus_queue.cc:1035] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:53.074486 21057 consensus_queue.cc:1035] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:53.077046 21057 raft_consensus.cc:2932] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:53.080714 12973 raft_consensus.cc:2932] T b8a350eade1c41e4b4c29e487374552f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:53.083942 12825 catalog_manager.cc:5484] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:53.088034 13365 raft_consensus.cc:2932] T b8a350eade1c41e4b4c29e487374552f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:12:53.096076 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet b8a350eade1c41e4b4c29e487374552f with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:53.096660 21361 tablet_replica.cc:330] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:53.096933 21361 raft_consensus.cc:2229] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:53.097208 21361 raft_consensus.cc:2258] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:53.097972 21361 ts_tablet_manager.cc:1875] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:53.103906 21361 ts_tablet_manager.cc:1888] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:53.104028 21361 log.cc:1196] T b8a350eade1c41e4b4c29e487374552f P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/b8a350eade1c41e4b4c29e487374552f
I20240501 08:12:53.104808 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet b8a350eade1c41e4b4c29e487374552f (table rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]) successfully deleted
I20240501 08:12:53.402776 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:53.419369 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:53.448732 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:53.498823 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:53.518266 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:53.551893 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:53.559610 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:53.649776 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:53.751449 13365 consensus_queue.cc:237] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:53.755069 13105 raft_consensus.cc:1261] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:53.755362 15756 raft_consensus.cc:1261] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:53.755534 19789 consensus_queue.cc:1035] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:53.755849 21153 consensus_queue.cc:1035] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:53.761718 21153 raft_consensus.cc:2932] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:53.762230 13105 raft_consensus.cc:2932] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:53.768626 12825 catalog_manager.cc:5484] T 3d16de310e7148768967be392b7af2ee P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:53.779048 15756 raft_consensus.cc:2932] T 3d16de310e7148768967be392b7af2ee P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:53.780364 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d16de310e7148768967be392b7af2ee with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:12:53.787902 21441 tablet_replica.cc:330] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:12:53.788209 21441 raft_consensus.cc:2229] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:53.788509 21441 raft_consensus.cc:2258] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:53.789186 21441 ts_tablet_manager.cc:1875] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:53.797274 21441 ts_tablet_manager.cc:1888] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:53.797619 21441 log.cc:1196] T 3d16de310e7148768967be392b7af2ee P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/3d16de310e7148768967be392b7af2ee
I20240501 08:12:53.799132 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 3d16de310e7148768967be392b7af2ee (table rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]) successfully deleted
I20240501 08:12:54.124219 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:54.156608 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:54.167395 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:54.188179 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:54.188197 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:54.230628 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:54.269784 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:54.289367 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:54.434650 16023 consensus_queue.cc:237] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:54.436681 13365 raft_consensus.cc:1261] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.436718 12973 raft_consensus.cc:1261] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:54.437181 20799 consensus_queue.cc:1035] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.437414 20547 consensus_queue.cc:1035] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:54.440809 20547 raft_consensus.cc:2932] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:54.441500 13365 raft_consensus.cc:2932] T d241e087e754485cb80080754d27aef7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:54.446740 12826 catalog_manager.cc:5484] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:54.453905 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet d241e087e754485cb80080754d27aef7 with delete_type TABLET_DATA_TOMBSTONED (TS 4d516916eeac468db514eeb928f06190 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:54.460259 12971 raft_consensus.cc:2932] T d241e087e754485cb80080754d27aef7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:54.476771 21527 tablet_replica.cc:330] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:54.477033 21527 raft_consensus.cc:2229] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:54.477290 21527 raft_consensus.cc:2258] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:54.477968 21527 ts_tablet_manager.cc:1875] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:54.484053 21527 ts_tablet_manager.cc:1888] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:54.484184 21527 log.cc:1196] T d241e087e754485cb80080754d27aef7 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/d241e087e754485cb80080754d27aef7
I20240501 08:12:54.484989 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet d241e087e754485cb80080754d27aef7 (table rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]) successfully deleted
I20240501 08:12:54.796116 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:54.824827 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:54.833825 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:54.848441 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:54.860231 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:54.865904 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:54.904745 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:54.944685 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:12:55.010298 12909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 47572.4us
W20240501 08:12:55.028662 12907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 170829us
I20240501 08:12:55.199038 15891 consensus_queue.cc:237] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:55.201292 16157 raft_consensus.cc:1261] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.202145 20660 consensus_queue.cc:1035] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.205001 20660 raft_consensus.cc:2932] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:55.205523 13234 raft_consensus.cc:1261] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:55.208513 21615 consensus_queue.cc:1035] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:55.208966 16157 raft_consensus.cc:2932] T 37b0020baddc4f47b625edc2e1180e51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:55.214141 13234 raft_consensus.cc:2932] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:55.228655 12828 catalog_manager.cc:5484] T 37b0020baddc4f47b625edc2e1180e51 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:55.249229 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b0020baddc4f47b625edc2e1180e51 with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:55.263833 21627 tablet_replica.cc:330] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:55.264073 21627 raft_consensus.cc:2229] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:55.264317 21627 raft_consensus.cc:2258] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:55.264932 21627 ts_tablet_manager.cc:1875] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:55.274258 21627 ts_tablet_manager.cc:1888] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:55.274394 21627 log.cc:1196] T 37b0020baddc4f47b625edc2e1180e51 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/37b0020baddc4f47b625edc2e1180e51
I20240501 08:12:55.275246 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 37b0020baddc4f47b625edc2e1180e51 (table rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]) successfully deleted
I20240501 08:12:55.561959 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:55.603247 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:55.670329 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:55.714522 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:55.726114 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:55.744827 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:55.747606 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:55.763967 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:55.955049 13234 consensus_queue.cc:237] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:55.957055 13103 raft_consensus.cc:1261] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:55.957074 16157 raft_consensus.cc:1261] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:55.957605 21063 consensus_queue.cc:1035] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:55.957861 21655 consensus_queue.cc:1035] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:55.960647 21655 raft_consensus.cc:2932] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:55.964738 13103 raft_consensus.cc:2932] T 53872527ad264dd9b4b1627ff8aa46db P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:55.971356 12833 catalog_manager.cc:5484] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:55.972158 16157 raft_consensus.cc:2932] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:55.991021 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 53872527ad264dd9b4b1627ff8aa46db with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:55.992043 21715 tablet_replica.cc:330] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:55.992266 21715 raft_consensus.cc:2229] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:55.992542 21715 raft_consensus.cc:2258] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:55.993196 21715 ts_tablet_manager.cc:1875] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:56.001448 21715 ts_tablet_manager.cc:1888] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:56.001649 21715 log.cc:1196] T 53872527ad264dd9b4b1627ff8aa46db P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/53872527ad264dd9b4b1627ff8aa46db
I20240501 08:12:56.002791 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 53872527ad264dd9b4b1627ff8aa46db (table rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]) successfully deleted
I20240501 08:12:56.214257 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:56.222307 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:56.223961 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:56.235993 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:56.246335 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:56.267253 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:56.304522 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:56.343812 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:56.491709 16023 consensus_queue.cc:237] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:12:56.497128 13363 raft_consensus.cc:1261] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:56.497745 20547 consensus_queue.cc:1035] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:56.500658 20547 raft_consensus.cc:2932] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:56.502063 12973 raft_consensus.cc:1261] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:56.502709 21772 consensus_queue.cc:1035] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:56.508131 13363 raft_consensus.cc:2932] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:56.515619 12831 catalog_manager.cc:5484] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:56.533908 12973 raft_consensus.cc:2932] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:12:56.567611 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet fc4ad07cecff43f0b867e4db444957e7 with delete_type TABLET_DATA_TOMBSTONED (TS 4d516916eeac468db514eeb928f06190 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44232
I20240501 08:12:56.577301 21800 tablet_replica.cc:330] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:12:56.577636 21800 raft_consensus.cc:2229] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:56.577920 21800 raft_consensus.cc:2258] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:56.579555 21800 ts_tablet_manager.cc:1875] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:56.595896 21800 ts_tablet_manager.cc:1888] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:56.596141 21800 log.cc:1196] T fc4ad07cecff43f0b867e4db444957e7 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/fc4ad07cecff43f0b867e4db444957e7
I20240501 08:12:56.597199 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet fc4ad07cecff43f0b867e4db444957e7 (table rebalancer_extra_table_10 [id=1ea038a9919c4a9fb5bf8203c2bf1e1d]) successfully deleted
I20240501 08:12:56.848227 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:56.850672 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:56.878202 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:56.893536 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:56.995587 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:57.054639 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:57.078696 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:57.087318 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:57.220129 13105 consensus_queue.cc:237] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:12:57.222344 13232 raft_consensus.cc:1261] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:57.222795 15891 raft_consensus.cc:1261] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:57.223300 21549 consensus_queue.cc:1035] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:57.223574 21152 consensus_queue.cc:1035] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:57.227145 21549 raft_consensus.cc:2932] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:57.231652 13232 raft_consensus.cc:2932] T 0f723c6321424530bcd41b9ecdcbe0b3 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:57.235680 15891 raft_consensus.cc:2932] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:57.236510 12831 catalog_manager.cc:5484] T 0f723c6321424530bcd41b9ecdcbe0b3 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:12:57.253835 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f723c6321424530bcd41b9ecdcbe0b3 with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:12:57.254679 21881 tablet_replica.cc:330] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:12:57.254933 21881 raft_consensus.cc:2229] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:57.255174 21881 raft_consensus.cc:2258] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:57.255844 21881 ts_tablet_manager.cc:1875] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:57.262367 21881 ts_tablet_manager.cc:1888] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:57.262552 21881 log.cc:1196] T 0f723c6321424530bcd41b9ecdcbe0b3 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/0f723c6321424530bcd41b9ecdcbe0b3
I20240501 08:12:57.263286 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 0f723c6321424530bcd41b9ecdcbe0b3 (table rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]) successfully deleted
I20240501 08:12:57.523406 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:57.526870 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:57.558221 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:57.560266 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:57.585278 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:57.590641 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:57.597893 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:57.639413 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:57.797677 13103 consensus_queue.cc:237] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:12:57.799809 16157 raft_consensus.cc:1261] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:57.799809 13363 raft_consensus.cc:1261] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:12:57.800412 21152 consensus_queue.cc:1035] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:57.802099 21549 consensus_queue.cc:1035] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:57.803212 21152 raft_consensus.cc:2932] T 13d9aa0c21e1405791e340cd79738256 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:57.804901 13363 raft_consensus.cc:2932] T 13d9aa0c21e1405791e340cd79738256 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:57.805420 16157 raft_consensus.cc:2932] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:57.813174 12826 catalog_manager.cc:5484] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:12:57.819286 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 13d9aa0c21e1405791e340cd79738256 with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57838
I20240501 08:12:57.819901 21969 tablet_replica.cc:330] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:12:57.820111 21969 raft_consensus.cc:2229] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:57.820395 21969 raft_consensus.cc:2258] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:57.821192 21969 ts_tablet_manager.cc:1875] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:57.828334 21969 ts_tablet_manager.cc:1888] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:12:57.828584 21969 log.cc:1196] T 13d9aa0c21e1405791e340cd79738256 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/13d9aa0c21e1405791e340cd79738256
I20240501 08:12:57.829452 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 13d9aa0c21e1405791e340cd79738256 (table rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]) successfully deleted
I20240501 08:12:58.161650 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:58.162389 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:58.162449 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:58.184267 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:58.202409 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:58.224887 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:58.250478 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:58.340911 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:58.429965 16024 consensus_queue.cc:237] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:12:58.432425 16157 raft_consensus.cc:1261] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:58.432466 13103 raft_consensus.cc:1261] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:12:58.432904 20799 consensus_queue.cc:1035] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:58.433176 20547 consensus_queue.cc:1035] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:12:58.435755 20799 raft_consensus.cc:2932] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:58.436729 13103 raft_consensus.cc:2932] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:58.437114 16157 raft_consensus.cc:2932] T 8953bcf510a2413cbceb4924f651b771 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:58.442584 12831 catalog_manager.cc:5484] T 8953bcf510a2413cbceb4924f651b771 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:12:58.452096 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 8953bcf510a2413cbceb4924f651b771 with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40166
I20240501 08:12:58.452697 22055 tablet_replica.cc:330] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:12:58.453107 22055 raft_consensus.cc:2229] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:12:58.453325 22055 raft_consensus.cc:2258] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:12:58.453910 22055 ts_tablet_manager.cc:1875] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:12:58.459698 22055 ts_tablet_manager.cc:1888] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:12:58.459908 22055 log.cc:1196] T 8953bcf510a2413cbceb4924f651b771 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/8953bcf510a2413cbceb4924f651b771
I20240501 08:12:58.460988 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 8953bcf510a2413cbceb4924f651b771 (table rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]) successfully deleted
I20240501 08:12:58.775295 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:58.785187 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:58.789673 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:58.805449 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:58.829604 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:12:58.837116 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:58.865437 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:58.865895 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:59.002315 13235 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ec6d272070b140d9b918d182613008f4"
dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49196
I20240501 08:12:59.002573 13235 raft_consensus.cc:595] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Received request to transfer leadership
I20240501 08:12:59.050485 21942 raft_consensus.cc:979] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: : Instructing follower 5dd722c770d34d6c9154a87917117161 to start an election
I20240501 08:12:59.050642 21942 raft_consensus.cc:1067] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Signalling peer 5dd722c770d34d6c9154a87917117161 to start an election
I20240501 08:12:59.052095 16024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ec6d272070b140d9b918d182613008f4"
dest_uuid: "5dd722c770d34d6c9154a87917117161"
from {username='slave'} at 127.6.56.67:56345
I20240501 08:12:59.052836 16024 raft_consensus.cc:482] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:12:59.052989 16024 raft_consensus.cc:3037] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:59.055660 16024 raft_consensus.cc:504] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:59.059041 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6d272070b140d9b918d182613008f4" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:12:59.059242 13365 raft_consensus.cc:3037] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:59.059404 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6d272070b140d9b918d182613008f4" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:12:59.059587 13232 raft_consensus.cc:3032] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:12:59.059664 13232 raft_consensus.cc:729] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:12:59.059957 13232 consensus_queue.cc:260] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:59.060315 13232 raft_consensus.cc:3037] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:59.061815 13365 raft_consensus.cc:2454] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 2.
I20240501 08:12:59.062685 13232 raft_consensus.cc:2454] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 2.
I20240501 08:12:59.063170 15959 leader_election.cc:302] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:12:59.063747 20547 raft_consensus.cc:2781] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:12:59.063865 20547 raft_consensus.cc:686] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 2 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:12:59.064088 20547 consensus_queue.cc:237] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:12:59.066370 13105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec6d272070b140d9b918d182613008f4" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:12:59.066558 13105 raft_consensus.cc:3037] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:12:59.072317 13105 raft_consensus.cc:2454] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 2.
I20240501 08:12:59.074152 16024 leader_election.cc:288] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:12:59.077855 12831 catalog_manager.cc:5484] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 1 to 2, leader changed from 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 2 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:12:59.231200 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:59.233770 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:59.237156 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:59.239440 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:12:59.299492 13213 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:59.327458 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:59.344142 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:59.434948 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:59.450652 13365 raft_consensus.cc:1261] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:59.451447 20547 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:59.590483 13103 raft_consensus.cc:1261] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:59.591081 20799 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:59.608556 13232 raft_consensus.cc:1261] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:12:59.609252 22199 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:12:59.861366 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:12:59.876735 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:12:59.887629 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:12:59.937289 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:12:59.961640 13213 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:12:59.970506 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:12:59.997834 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:00.036449 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:00.145406 15756 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e"
dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47002
I20240501 08:13:00.145676 15756 raft_consensus.cc:595] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:00.275662 22266 raft_consensus.cc:979] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: : Instructing follower 6ed448d04e4646658c0de37d829ff15b to start an election
I20240501 08:13:00.275815 22266 raft_consensus.cc:1067] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Signalling peer 6ed448d04e4646658c0de37d829ff15b to start an election
I20240501 08:13:00.277050 13235 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e"
dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
from {username='slave'} at 127.6.56.69:57331
I20240501 08:13:00.277283 13235 raft_consensus.cc:482] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:00.277386 13235 raft_consensus.cc:3037] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:00.279525 13235 raft_consensus.cc:504] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:00.290634 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:13:00.290853 16024 raft_consensus.cc:3037] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:00.294974 16024 raft_consensus.cc:2454] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 2.
I20240501 08:13:00.297132 15756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:13:00.297348 15756 raft_consensus.cc:3032] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:00.297425 15756 raft_consensus.cc:729] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:13:00.297714 15756 consensus_queue.cc:260] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:00.298203 15756 raft_consensus.cc:3037] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:00.299099 13235 leader_election.cc:288] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:13:00.299669 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:13:00.299890 13103 raft_consensus.cc:3037] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:00.302832 15756 raft_consensus.cc:2454] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 2.
I20240501 08:13:00.303634 13170 leader_election.cc:302] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:13:00.303907 22112 raft_consensus.cc:2781] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:00.310039 13103 raft_consensus.cc:2454] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 2.
I20240501 08:13:00.326742 22112 raft_consensus.cc:686] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 2 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:13:00.327303 22112 consensus_queue.cc:237] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:00.354486 12828 catalog_manager.cc:5484] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 1 to 2, leader changed from 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 2 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:00.500931 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:00.518149 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:00.573910 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:00.607299 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:00.629832 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:00.655963 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:00.719039 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:00.730008 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:00.791532 15753 raft_consensus.cc:1261] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:00.792026 22292 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:00.808962 16022 raft_consensus.cc:1261] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:00.809489 22292 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:00.817490 13105 raft_consensus.cc:1261] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:00.821169 22292 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:01.143337 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:01.147315 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:01.164690 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:01.192785 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:01.194959 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:01.199545 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:01.208547 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:01.217481 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:01.394762 13232 consensus_queue.cc:237] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:01.396790 16154 raft_consensus.cc:1261] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:01.397536 13365 raft_consensus.cc:1261] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:01.398528 22293 consensus_queue.cc:1035] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:01.399091 22293 consensus_queue.cc:1035] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:01.401154 22293 raft_consensus.cc:2932] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:01.402545 13365 raft_consensus.cc:2932] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:01.405037 16154 raft_consensus.cc:2932] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:01.409216 12831 catalog_manager.cc:5484] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:01.419533 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet e6fefd76544642ddb787cc6febe8df82 with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57838
I20240501 08:13:01.422534 22435 tablet_replica.cc:330] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:13:01.422787 22435 raft_consensus.cc:2229] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:01.423013 22435 raft_consensus.cc:2258] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:01.423579 22435 ts_tablet_manager.cc:1875] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:01.430711 22435 ts_tablet_manager.cc:1888] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:13:01.430883 22435 log.cc:1196] T e6fefd76544642ddb787cc6febe8df82 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/e6fefd76544642ddb787cc6febe8df82
I20240501 08:13:01.431721 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet e6fefd76544642ddb787cc6febe8df82 (table rebalance_test_table_5 [id=0e717b4ed14d4b91858e6cdd752f1662]) successfully deleted
I20240501 08:13:01.710767 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:01.729641 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:01.747344 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:01.770694 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:01.802402 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:01.813629 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:01.835767 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:01.880053 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:02.031678 15892 consensus_queue.cc:237] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:02.033972 13105 raft_consensus.cc:1261] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:02.034499 13232 raft_consensus.cc:1261] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:02.034559 22335 consensus_queue.cc:1035] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:02.035524 22335 consensus_queue.cc:1035] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:02.038050 20660 raft_consensus.cc:2932] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:02.038693 13105 raft_consensus.cc:2932] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:02.039522 13232 raft_consensus.cc:2932] T 8c487b2c7fe24515a8970a07883c126f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:02.045428 12831 catalog_manager.cc:5484] T 8c487b2c7fe24515a8970a07883c126f P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:02.062945 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c487b2c7fe24515a8970a07883c126f with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40166
I20240501 08:13:02.069355 22524 tablet_replica.cc:330] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:13:02.069553 22524 raft_consensus.cc:2229] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.069808 22524 raft_consensus.cc:2258] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.070462 22524 ts_tablet_manager.cc:1875] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.077289 22524 ts_tablet_manager.cc:1888] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:02.077481 22524 log.cc:1196] T 8c487b2c7fe24515a8970a07883c126f P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/8c487b2c7fe24515a8970a07883c126f
I20240501 08:13:02.078378 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 8c487b2c7fe24515a8970a07883c126f (table rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]) successfully deleted
I20240501 08:13:02.395941 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:02.433604 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:02.437371 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:02.459542 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:02.460757 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:02.485148 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:02.519766 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:02.565275 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:02.675205 15892 consensus_queue.cc:237] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:02.689391 13232 raft_consensus.cc:1261] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:02.689605 12971 raft_consensus.cc:1261] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:02.690402 22335 consensus_queue.cc:1035] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:02.690912 22335 consensus_queue.cc:1035] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:02.693607 22335 raft_consensus.cc:2932] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:02.697062 13232 raft_consensus.cc:2932] T fa10a6f53f7f48cf9410c80e1056439d P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:02.698951 12971 raft_consensus.cc:2932] T fa10a6f53f7f48cf9410c80e1056439d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:02.700423 12830 catalog_manager.cc:5484] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:02.710884 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet fa10a6f53f7f48cf9410c80e1056439d with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:13:02.711510 22609 tablet_replica.cc:330] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:13:02.711808 22609 raft_consensus.cc:2229] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:02.712116 22609 raft_consensus.cc:2258] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:02.713022 22609 ts_tablet_manager.cc:1875] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:02.719916 22609 ts_tablet_manager.cc:1888] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:02.720050 22609 log.cc:1196] T fa10a6f53f7f48cf9410c80e1056439d P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/fa10a6f53f7f48cf9410c80e1056439d
I20240501 08:13:02.720784 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet fa10a6f53f7f48cf9410c80e1056439d (table rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]) successfully deleted
I20240501 08:13:03.010593 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:03.022864 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:03.070822 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:03.085268 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:03.117535 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:03.147006 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:03.154965 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:03.172243 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:03.300189 15753 consensus_queue.cc:237] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:03.302340 13103 raft_consensus.cc:1261] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:03.302582 12971 raft_consensus.cc:1261] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:03.302979 20877 consensus_queue.cc:1035] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:03.303213 22557 consensus_queue.cc:1035] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:03.307008 22557 raft_consensus.cc:2932] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:03.307828 12971 raft_consensus.cc:2932] T 321bf6596ff24ec7af79e0510eadedce P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:03.315143 12825 catalog_manager.cc:5484] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:03.323350 13105 raft_consensus.cc:2932] T 321bf6596ff24ec7af79e0510eadedce P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:03.347175 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 321bf6596ff24ec7af79e0510eadedce with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:13:03.347797 22701 tablet_replica.cc:330] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:13:03.348132 22701 raft_consensus.cc:2229] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:03.348873 22701 raft_consensus.cc:2258] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:03.349625 22701 ts_tablet_manager.cc:1875] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:03.356047 22701 ts_tablet_manager.cc:1888] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:03.356242 22701 log.cc:1196] T 321bf6596ff24ec7af79e0510eadedce P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/321bf6596ff24ec7af79e0510eadedce
I20240501 08:13:03.356886 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 321bf6596ff24ec7af79e0510eadedce (table rebalancer_extra_table_34 [id=c33d8b65cab4442eac81e761d2535a86]) successfully deleted
I20240501 08:13:03.745580 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:03.748207 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:03.757393 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:03.760632 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:03.778895 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:03.800808 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:03.803654 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:03.833609 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:04.048728 16022 consensus_queue.cc:237] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:04.051602 13105 raft_consensus.cc:1261] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:13:04.052168 22334 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:04.052347 13365 raft_consensus.cc:1261] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:13:04.053012 22334 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:13:04.054960 20547 raft_consensus.cc:2932] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:04.055589 13105 raft_consensus.cc:2932] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:04.057582 13365 raft_consensus.cc:2932] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:04.061902 12830 catalog_manager.cc:5484] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 4 to 6, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 2 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:04.077852 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet ec6d272070b140d9b918d182613008f4 with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34524
I20240501 08:13:04.078399 22842 tablet_replica.cc:330] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:13:04.078655 22842 raft_consensus.cc:2229] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:04.079106 22842 raft_consensus.cc:2258] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:04.079684 22842 ts_tablet_manager.cc:1875] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:04.086299 22842 ts_tablet_manager.cc:1888] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240501 08:13:04.086475 22842 log.cc:1196] T ec6d272070b140d9b918d182613008f4 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/ec6d272070b140d9b918d182613008f4
I20240501 08:13:04.087363 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet ec6d272070b140d9b918d182613008f4 (table rebalance_test_table_7 [id=ab73f40ad68f450ca7d155c31949855f]) successfully deleted
I20240501 08:13:04.307278 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:04.336855 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:04.352352 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:04.366734 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:04.380127 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:04.484476 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:04.513270 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:04.516013 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240501 08:13:04.640108 15959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.66:35509, user_credentials={real_user=slave}} blocked reactor thread for 49593.4us
I20240501 08:13:04.699144 13234 consensus_queue.cc:237] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:04.701061 13105 raft_consensus.cc:1261] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:04.701172 16023 raft_consensus.cc:1261] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:04.701558 22819 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:04.701961 22820 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:04.704519 22819 raft_consensus.cc:2932] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:04.707170 13105 raft_consensus.cc:2932] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:04.708489 16023 raft_consensus.cc:2932] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:04.715441 12825 catalog_manager.cc:5484] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 2 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:04.729753 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aa998326f3c4ced9afbee7e54bffa0e with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40166
I20240501 08:13:04.736647 22923 tablet_replica.cc:330] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:13:04.736874 22923 raft_consensus.cc:2229] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:04.737054 22923 raft_consensus.cc:2258] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:04.737738 22923 ts_tablet_manager.cc:1875] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:04.743969 22923 ts_tablet_manager.cc:1888] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 08:13:04.744136 22923 log.cc:1196] T 6aa998326f3c4ced9afbee7e54bffa0e P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/6aa998326f3c4ced9afbee7e54bffa0e
I20240501 08:13:04.745044 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 6aa998326f3c4ced9afbee7e54bffa0e (table rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]) successfully deleted
I20240501 08:13:04.802117 13365 consensus_queue.cc:237] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:04.804924 15892 raft_consensus.cc:1261] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:04.805195 16154 raft_consensus.cc:1261] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:04.805444 13301 consensus_peers.cc:485] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6154bf2fadb45e58c27c46165b5824f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:04.805878 22375 consensus_queue.cc:1035] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:04.806155 22175 consensus_queue.cc:1035] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:04.809466 22375 raft_consensus.cc:2932] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:04.810065 15892 raft_consensus.cc:2932] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:04.815204 16154 raft_consensus.cc:2932] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:04.816033 12824 catalog_manager.cc:5484] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:04.862108 13234 consensus_queue.cc:237] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
W20240501 08:13:04.864472 13168 consensus_peers.cc:485] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5741e5458ccd4af6a3972495ccf40b8f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:04.864498 13105 raft_consensus.cc:1261] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:04.864974 22819 consensus_queue.cc:1035] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:04.865232 13365 raft_consensus.cc:1261] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:04.865679 22819 consensus_queue.cc:1035] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:04.868245 22820 raft_consensus.cc:2932] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:04.868898 13365 raft_consensus.cc:2932] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:04.876811 12825 catalog_manager.cc:5484] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:04.880663 13105 raft_consensus.cc:2932] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:04.995412 13105 consensus_queue.cc:237] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:05.001256 15891 raft_consensus.cc:1261] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.001534 13234 raft_consensus.cc:1261] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:05.002032 13040 consensus_peers.cc:485] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c99eddd74b7740dcab6b93ca437b51ae. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.002377 22392 consensus_queue.cc:1035] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.002951 22392 consensus_queue.cc:1035] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.005899 22993 raft_consensus.cc:2932] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:05.006520 13234 raft_consensus.cc:2932] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:05.010582 15891 raft_consensus.cc:2932] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:05.013492 12832 catalog_manager.cc:5484] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:05.058933 13103 consensus_queue.cc:237] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:05.061385 12971 raft_consensus.cc:1261] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:13:05.061539 13038 consensus_peers.cc:485] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a412b8ae0aa648d7979fac58b50d86a6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.062115 13363 raft_consensus.cc:1261] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:05.062391 22993 consensus_queue.cc:1035] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:05.062685 22392 consensus_queue.cc:1035] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:05.065896 22993 raft_consensus.cc:2932] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:05.066412 13363 raft_consensus.cc:2932] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:05.069324 12971 raft_consensus.cc:2932] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:05.071228 12825 catalog_manager.cc:5484] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:05.111410 13103 consensus_queue.cc:237] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:05.113785 13234 raft_consensus.cc:1261] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:05.113945 13363 raft_consensus.cc:1261] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:13:05.113996 13038 consensus_peers.cc:485] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99ecb2ec0b3d4fd7a02896cc9a28ba27. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.114332 22674 consensus_queue.cc:1035] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:05.114593 22993 consensus_queue.cc:1035] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:05.118362 22993 raft_consensus.cc:2932] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:05.120575 13363 raft_consensus.cc:2932] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:05.122088 13234 raft_consensus.cc:2932] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:05.124384 12832 catalog_manager.cc:5484] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:05.167742 15891 consensus_queue.cc:237] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:05.170301 15753 raft_consensus.cc:1261] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:05.170612 15825 consensus_peers.cc:485] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8203d209d624d2695e5b32bbfa0edf7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.170692 16023 raft_consensus.cc:1261] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.171375 23049 consensus_queue.cc:1035] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.171677 22335 consensus_queue.cc:1035] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.174834 22335 raft_consensus.cc:2932] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:05.178362 15753 raft_consensus.cc:2932] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:05.183912 12824 catalog_manager.cc:5484] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:05.184159 16023 raft_consensus.cc:2932] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:05.232589 16023 consensus_queue.cc:237] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:05.235256 15891 raft_consensus.cc:1261] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:05.235539 15960 consensus_peers.cc:485] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e61276f6f4c40f8987747b997ac52ac. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.235818 13234 raft_consensus.cc:1261] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.235874 20547 consensus_queue.cc:1035] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.236465 22334 consensus_queue.cc:1035] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.240957 22334 raft_consensus.cc:2932] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:05.243952 13232 raft_consensus.cc:2932] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:05.246331 15891 raft_consensus.cc:2932] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:05.248097 12825 catalog_manager.cc:5484] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:05.281031 23086 ts_tablet_manager.cc:902] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:13:05.285274 15891 consensus_queue.cc:237] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
W20240501 08:13:05.290257 15825 consensus_peers.cc:485] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85056837a567420a8b480b34b6017b40. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.291318 15753 raft_consensus.cc:1261] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.294605 22335 consensus_queue.cc:1035] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.296641 23086 tablet_copy_client.cc:308] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:13:05.297986 16153 raft_consensus.cc:1261] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.298682 13254 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet 5741e5458ccd4af6a3972495ccf40b8f from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:54825)
I20240501 08:13:05.298854 13254 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet 5741e5458ccd4af6a3972495ccf40b8f from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:54825: session id = 6cc32f6e2d49477f9a23ce6e444039fb-5741e5458ccd4af6a3972495ccf40b8f
I20240501 08:13:05.300160 23049 consensus_queue.cc:1035] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.301656 13254 tablet_copy_source_session.cc:205] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.303040 23086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5741e5458ccd4af6a3972495ccf40b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:05.303354 22335 raft_consensus.cc:2932] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:05.304903 16153 raft_consensus.cc:2932] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:05.311337 23086 tablet_copy_client.cc:783] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:05.311607 23086 tablet_copy_client.cc:647] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:05.313540 23086 tablet_copy_client.cc:523] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:05.313271 12832 catalog_manager.cc:5484] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:05.313416 15753 raft_consensus.cc:2932] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:05.317644 23086 tablet_bootstrap.cc:492] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:13:05.379593 23086 tablet_bootstrap.cc:492] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:05.380026 23086 tablet_bootstrap.cc:492] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:13:05.380264 23086 ts_tablet_manager.cc:1367] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.063s user 0.045s sys 0.009s
I20240501 08:13:05.380864 23086 raft_consensus.cc:348] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:05.381042 23086 raft_consensus.cc:729] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:13:05.381247 23086 consensus_queue.cc:260] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:05.381700 23086 ts_tablet_manager.cc:1398] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:13:05.382692 13254 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-5741e5458ccd4af6a3972495ccf40b8f received from {username='slave'} at 127.6.56.72:54825
I20240501 08:13:05.382822 13254 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-5741e5458ccd4af6a3972495ccf40b8f on tablet 5741e5458ccd4af6a3972495ccf40b8f with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:13:05.389729 16153 consensus_queue.cc:237] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:05.392472 15753 raft_consensus.cc:1261] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.392657 15891 raft_consensus.cc:1261] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.393067 22978 consensus_queue.cc:1035] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:05.393348 16092 consensus_peers.cc:485] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cabfd31e8ee941cf9f69bc04a0510d11. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.393357 22876 consensus_queue.cc:1035] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.396248 22978 raft_consensus.cc:2932] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:05.396965 15753 raft_consensus.cc:2932] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:05.403822 12832 catalog_manager.cc:5484] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:05.407248 15891 raft_consensus.cc:2932] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:05.410759 23120 ts_tablet_manager.cc:902] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:05.430050 23120 tablet_copy_client.cc:308] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:05.430748 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet a6154bf2fadb45e58c27c46165b5824f from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:39327)
I20240501 08:13:05.430886 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet a6154bf2fadb45e58c27c46165b5824f from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:39327: session id = 76d830e199eb461c8ca689c9460ee0c2-a6154bf2fadb45e58c27c46165b5824f
I20240501 08:13:05.433570 13385 tablet_copy_source_session.cc:205] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.435016 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6154bf2fadb45e58c27c46165b5824f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:05.441540 23120 tablet_copy_client.cc:783] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:05.442106 23120 tablet_copy_client.cc:647] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:05.444423 23120 tablet_copy_client.cc:523] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:05.447194 23120 tablet_bootstrap.cc:492] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:05.449795 23136 ts_tablet_manager.cc:902] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:13:05.454169 23120 tablet_bootstrap.cc:492] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:05.454551 23120 tablet_bootstrap.cc:492] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:05.454797 23120 ts_tablet_manager.cc:1367] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240501 08:13:05.455543 23120 raft_consensus.cc:348] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:05.455763 23120 raft_consensus.cc:729] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:05.455976 23120 consensus_queue.cc:260] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:05.457653 23120 ts_tablet_manager.cc:1398] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:05.457968 16153 consensus_queue.cc:237] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:05.459510 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-a6154bf2fadb45e58c27c46165b5824f received from {username='slave'} at 127.6.56.65:39327
I20240501 08:13:05.459614 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-a6154bf2fadb45e58c27c46165b5824f on tablet a6154bf2fadb45e58c27c46165b5824f with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:13:05.461756 13103 raft_consensus.cc:1261] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.462265 12974 raft_consensus.cc:1261] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.462702 22876 consensus_queue.cc:1035] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:05.462999 16093 consensus_peers.cc:485] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cc25959b69a41c98d196ed88117b26f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.463230 22978 consensus_queue.cc:1035] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.487777 22978 raft_consensus.cc:2932] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:05.488938 12974 raft_consensus.cc:2932] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:05.490236 23136 tablet_copy_client.cc:308] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:13:05.490911 13125 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet c99eddd74b7740dcab6b93ca437b51ae from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:40479)
I20240501 08:13:05.491060 13125 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet c99eddd74b7740dcab6b93ca437b51ae from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:40479: session id = 982f49fc87f149439cda0de2261b6bfd-c99eddd74b7740dcab6b93ca437b51ae
I20240501 08:13:05.493806 13125 tablet_copy_source_session.cc:205] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.494074 13103 raft_consensus.cc:2932] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:05.496464 12825 catalog_manager.cc:5484] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:05.496910 23136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c99eddd74b7740dcab6b93ca437b51ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:05.511047 23136 tablet_copy_client.cc:783] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:05.511315 23136 tablet_copy_client.cc:647] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:05.513506 23136 tablet_copy_client.cc:523] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:05.518131 23136 tablet_bootstrap.cc:492] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:13:05.522670 23145 ts_tablet_manager.cc:902] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:13:05.554061 23136 tablet_bootstrap.cc:492] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:05.554490 23136 tablet_bootstrap.cc:492] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:13:05.554754 23136 ts_tablet_manager.cc:1367] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.004s
I20240501 08:13:05.555476 23136 raft_consensus.cc:348] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:05.555698 23136 raft_consensus.cc:729] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:13:05.555922 23136 consensus_queue.cc:260] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:05.556535 23136 ts_tablet_manager.cc:1398] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:13:05.557308 13125 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-c99eddd74b7740dcab6b93ca437b51ae received from {username='slave'} at 127.6.56.69:40479
I20240501 08:13:05.557440 13125 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-c99eddd74b7740dcab6b93ca437b51ae on tablet c99eddd74b7740dcab6b93ca437b51ae with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:13:05.587646 23086 ts_tablet_manager.cc:902] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:13:05.615157 23086 tablet_copy_client.cc:308] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:13:05.617197 13125 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:42629)
I20240501 08:13:05.617403 13125 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:42629: session id = 6cc32f6e2d49477f9a23ce6e444039fb-99ecb2ec0b3d4fd7a02896cc9a28ba27
I20240501 08:13:05.618373 23145 tablet_copy_client.cc:308] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:13:05.618901 13124 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet a412b8ae0aa648d7979fac58b50d86a6 from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:47641)
I20240501 08:13:05.619040 13124 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet a412b8ae0aa648d7979fac58b50d86a6 from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:47641: session id = 4d516916eeac468db514eeb928f06190-a412b8ae0aa648d7979fac58b50d86a6
I20240501 08:13:05.620282 13125 tablet_copy_source_session.cc:205] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.617388 23158 ts_tablet_manager.cc:902] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:13:05.621615 13124 tablet_copy_source_session.cc:205] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.623481 23086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:05.628734 23086 tablet_copy_client.cc:783] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:05.629956 23086 tablet_copy_client.cc:647] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:05.631778 23145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a412b8ae0aa648d7979fac58b50d86a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:05.632531 23086 tablet_copy_client.cc:523] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:05.639150 23086 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:13:05.648623 23158 tablet_copy_client.cc:308] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:13:05.648649 23145 tablet_copy_client.cc:783] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:05.648936 23145 tablet_copy_client.cc:647] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:05.651160 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet f8203d209d624d2695e5b32bbfa0edf7 from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:39283)
I20240501 08:13:05.651300 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet f8203d209d624d2695e5b32bbfa0edf7 from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:39283: session id = e3fefd4dca2d41678187047f06c10187-f8203d209d624d2695e5b32bbfa0edf7
I20240501 08:13:05.651779 23145 tablet_copy_client.cc:523] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:05.654455 15913 tablet_copy_source_session.cc:205] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.658461 23145 tablet_bootstrap.cc:492] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:13:05.672116 23158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8203d209d624d2695e5b32bbfa0edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:05.678725 23158 tablet_copy_client.cc:783] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:05.678989 23158 tablet_copy_client.cc:647] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:05.698529 23158 tablet_copy_client.cc:523] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:05.703541 23158 tablet_bootstrap.cc:492] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:13:05.706698 23173 ts_tablet_manager.cc:902] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:13:05.717247 23158 tablet_bootstrap.cc:492] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:05.724265 23158 tablet_bootstrap.cc:492] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:13:05.724591 23158 ts_tablet_manager.cc:1367] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.001s
I20240501 08:13:05.724839 16153 raft_consensus.cc:1203] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:13:05.727386 23158 raft_consensus.cc:348] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:05.727650 23158 raft_consensus.cc:729] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:13:05.728991 23158 consensus_queue.cc:260] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:05.738323 23158 ts_tablet_manager.cc:1398] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240501 08:13:05.738968 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-f8203d209d624d2695e5b32bbfa0edf7 received from {username='slave'} at 127.6.56.66:39283
I20240501 08:13:05.739087 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session e3fefd4dca2d41678187047f06c10187-f8203d209d624d2695e5b32bbfa0edf7 on tablet f8203d209d624d2695e5b32bbfa0edf7 with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:13:05.745368 23173 tablet_copy_client.cc:308] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:13:05.746140 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 85056837a567420a8b480b34b6017b40 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:37921)
I20240501 08:13:05.746292 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 85056837a567420a8b480b34b6017b40 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:37921: session id = 4ecb724cb4764a53988fde00407db4c0-85056837a567420a8b480b34b6017b40
I20240501 08:13:05.747730 13232 consensus_queue.cc:237] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:05.750881 23145 tablet_bootstrap.cc:492] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:05.751332 23145 tablet_bootstrap.cc:492] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:13:05.751415 15913 tablet_copy_source_session.cc:205] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.751616 23145 ts_tablet_manager.cc:1367] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.093s user 0.060s sys 0.011s
I20240501 08:13:05.752035 23086 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:05.752533 23086 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:13:05.752503 23145 raft_consensus.cc:348] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:05.752722 23145 raft_consensus.cc:729] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:13:05.752801 23086 ts_tablet_manager.cc:1367] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.114s user 0.049s sys 0.003s
I20240501 08:13:05.753024 23173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85056837a567420a8b480b34b6017b40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:05.752950 23145 consensus_queue.cc:260] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:05.753583 23145 ts_tablet_manager.cc:1398] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 08:13:05.754436 13124 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-a412b8ae0aa648d7979fac58b50d86a6 received from {username='slave'} at 127.6.56.70:47641
I20240501 08:13:05.754556 13124 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 4d516916eeac468db514eeb928f06190-a412b8ae0aa648d7979fac58b50d86a6 on tablet a412b8ae0aa648d7979fac58b50d86a6 with peer 4d516916eeac468db514eeb928f06190
I20240501 08:13:05.760915 23173 tablet_copy_client.cc:783] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:05.761165 23173 tablet_copy_client.cc:647] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
W20240501 08:13:05.761298 13171 consensus_peers.cc:485] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08106d7d42674f01bbee94b6236fb9f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.761585 13365 raft_consensus.cc:1261] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.759929 23086 raft_consensus.cc:348] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:05.761849 23086 raft_consensus.cc:729] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:13:05.762146 23086 consensus_queue.cc:260] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:05.762317 13103 raft_consensus.cc:1261] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.762851 23086 ts_tablet_manager.cc:1398] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 08:13:05.762974 22819 consensus_queue.cc:1035] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.763532 22819 consensus_queue.cc:1035] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.766867 13124 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-99ecb2ec0b3d4fd7a02896cc9a28ba27 received from {username='slave'} at 127.6.56.72:42629
I20240501 08:13:05.766984 13124 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-99ecb2ec0b3d4fd7a02896cc9a28ba27 on tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:13:05.770939 23173 tablet_copy_client.cc:523] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:05.774595 22973 raft_consensus.cc:2932] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:05.775142 13365 raft_consensus.cc:2932] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:05.775612 23173 tablet_bootstrap.cc:492] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:05.783710 13103 raft_consensus.cc:2932] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:05.795262 23173 tablet_bootstrap.cc:492] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:05.795152 12825 catalog_manager.cc:5484] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:05.795642 23173 tablet_bootstrap.cc:492] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:05.796339 23173 ts_tablet_manager.cc:1367] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.012s
I20240501 08:13:05.797078 23173 raft_consensus.cc:348] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:05.797793 23173 raft_consensus.cc:729] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:05.798061 23173 consensus_queue.cc:260] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:05.798768 23173 ts_tablet_manager.cc:1398] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:13:05.800334 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-85056837a567420a8b480b34b6017b40 received from {username='slave'} at 127.6.56.68:37921
I20240501 08:13:05.800465 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-85056837a567420a8b480b34b6017b40 on tablet 85056837a567420a8b480b34b6017b40 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:05.815526 23120 ts_tablet_manager.cc:902] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:05.816977 23120 tablet_copy_client.cc:308] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:13:05.817590 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 5e61276f6f4c40f8987747b997ac52ac from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:52501)
I20240501 08:13:05.817728 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 5e61276f6f4c40f8987747b997ac52ac from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:52501: session id = 76d830e199eb461c8ca689c9460ee0c2-5e61276f6f4c40f8987747b997ac52ac
I20240501 08:13:05.819976 16045 tablet_copy_source_session.cc:205] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.821194 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e61276f6f4c40f8987747b997ac52ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:05.827526 23120 tablet_copy_client.cc:783] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:05.827760 23120 tablet_copy_client.cc:647] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:05.829456 23120 tablet_copy_client.cc:523] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:05.833287 23120 tablet_bootstrap.cc:492] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:05.838485 23120 tablet_bootstrap.cc:492] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:05.838853 23120 tablet_bootstrap.cc:492] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:05.839068 23120 ts_tablet_manager.cc:1367] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:13:05.839709 23120 raft_consensus.cc:348] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:05.839919 23120 raft_consensus.cc:729] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:05.840171 23120 consensus_queue.cc:260] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:05.840837 23120 ts_tablet_manager.cc:1398] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:13:05.841490 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-5e61276f6f4c40f8987747b997ac52ac received from {username='slave'} at 127.6.56.65:52501
I20240501 08:13:05.841589 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-5e61276f6f4c40f8987747b997ac52ac on tablet 5e61276f6f4c40f8987747b997ac52ac with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:13:05.854507 12974 consensus_queue.cc:237] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:05.856884 15891 raft_consensus.cc:1261] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:05.857035 15753 raft_consensus.cc:1261] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:05.857231 12909 consensus_peers.cc:485] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34a5baf3631b45d68a2003fc4539f799. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:05.857523 22897 consensus_queue.cc:1035] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.857754 22077 consensus_queue.cc:1035] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:05.865365 22077 raft_consensus.cc:2932] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:05.866082 15891 raft_consensus.cc:2932] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:05.873026 12824 catalog_manager.cc:5484] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:05.877216 15753 raft_consensus.cc:2932] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:05.903198 23216 ts_tablet_manager.cc:902] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:13:05.929984 12974 raft_consensus.cc:1203] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:05.938205 23216 tablet_copy_client.cc:308] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.72:45383
I20240501 08:13:05.939054 16178 tablet_copy_service.cc:140] P 6cc32f6e2d49477f9a23ce6e444039fb: Received BeginTabletCopySession request for tablet 8cc25959b69a41c98d196ed88117b26f from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:42551)
I20240501 08:13:05.939213 16178 tablet_copy_service.cc:161] P 6cc32f6e2d49477f9a23ce6e444039fb: Beginning new tablet copy session on tablet 8cc25959b69a41c98d196ed88117b26f from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:42551: session id = 6ed448d04e4646658c0de37d829ff15b-8cc25959b69a41c98d196ed88117b26f
I20240501 08:13:05.941548 16178 tablet_copy_source_session.cc:205] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.943676 23216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc25959b69a41c98d196ed88117b26f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:05.959190 23216 tablet_copy_client.cc:783] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:05.959439 23216 tablet_copy_client.cc:647] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:05.961302 23216 tablet_copy_client.cc:523] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:05.964077 23216 tablet_bootstrap.cc:492] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:13:05.970402 23216 tablet_bootstrap.cc:492] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:05.970793 23216 tablet_bootstrap.cc:492] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:13:05.971055 23216 ts_tablet_manager.cc:1367] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240501 08:13:05.972124 23216 raft_consensus.cc:348] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:05.972373 23216 raft_consensus.cc:729] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:13:05.972575 23216 consensus_queue.cc:260] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:05.973124 23216 ts_tablet_manager.cc:1398] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:05.973884 16178 tablet_copy_service.cc:342] P 6cc32f6e2d49477f9a23ce6e444039fb: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-8cc25959b69a41c98d196ed88117b26f received from {username='slave'} at 127.6.56.67:42551
I20240501 08:13:05.974011 16178 tablet_copy_service.cc:433] P 6cc32f6e2d49477f9a23ce6e444039fb: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-8cc25959b69a41c98d196ed88117b26f on tablet 8cc25959b69a41c98d196ed88117b26f with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:13:05.981554 23225 ts_tablet_manager.cc:902] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:13:05.984969 23225 tablet_copy_client.cc:308] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.72:45383
I20240501 08:13:05.985899 16178 tablet_copy_service.cc:140] P 6cc32f6e2d49477f9a23ce6e444039fb: Received BeginTabletCopySession request for tablet cabfd31e8ee941cf9f69bc04a0510d11 from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:38083)
I20240501 08:13:05.986091 16178 tablet_copy_service.cc:161] P 6cc32f6e2d49477f9a23ce6e444039fb: Beginning new tablet copy session on tablet cabfd31e8ee941cf9f69bc04a0510d11 from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:38083: session id = 5dd722c770d34d6c9154a87917117161-cabfd31e8ee941cf9f69bc04a0510d11
I20240501 08:13:05.989945 16178 tablet_copy_source_session.cc:205] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:05.991438 23225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cabfd31e8ee941cf9f69bc04a0510d11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:06.000607 23225 tablet_copy_client.cc:783] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:06.001016 23225 tablet_copy_client.cc:647] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:06.010026 23225 tablet_copy_client.cc:523] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:06.015084 23225 tablet_bootstrap.cc:492] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:13:06.023334 23225 tablet_bootstrap.cc:492] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:06.023727 23225 tablet_bootstrap.cc:492] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:13:06.024036 23225 ts_tablet_manager.cc:1367] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20240501 08:13:06.024765 23225 raft_consensus.cc:348] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:06.024947 23225 raft_consensus.cc:729] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:13:06.025305 23225 consensus_queue.cc:260] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:06.025755 23225 ts_tablet_manager.cc:1398] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:06.027858 16178 tablet_copy_service.cc:342] P 6cc32f6e2d49477f9a23ce6e444039fb: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-cabfd31e8ee941cf9f69bc04a0510d11 received from {username='slave'} at 127.6.56.71:38083
I20240501 08:13:06.027985 16178 tablet_copy_service.cc:433] P 6cc32f6e2d49477f9a23ce6e444039fb: ending tablet copy session 5dd722c770d34d6c9154a87917117161-cabfd31e8ee941cf9f69bc04a0510d11 on tablet cabfd31e8ee941cf9f69bc04a0510d11 with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:13:06.034430 15754 raft_consensus.cc:1203] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:06.041361 15891 raft_consensus.cc:1203] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:13:06.052345 15754 consensus_queue.cc:237] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:06.054944 16023 raft_consensus.cc:1261] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:06.055245 13232 raft_consensus.cc:1261] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:06.055740 23212 consensus_queue.cc:1035] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:06.057567 15689 consensus_peers.cc:485] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba08236c71df4c8190f6b7644ed944ed. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:06.060516 23212 consensus_queue.cc:1035] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:06.063524 23212 raft_consensus.cc:2932] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:06.064734 13232 raft_consensus.cc:2932] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:06.068564 16023 raft_consensus.cc:2932] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:06.071045 12824 catalog_manager.cc:5484] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 2, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:06.082150 13365 raft_consensus.cc:1203] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:06.125926 13103 raft_consensus.cc:1203] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:06.129312 13365 consensus_queue.cc:237] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:06.131858 16153 raft_consensus.cc:1261] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:06.132002 12974 raft_consensus.cc:1261] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:06.132051 13301 consensus_peers.cc:485] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f22fc6797cb24451b78a5c1e5f6306ba. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:06.132514 22175 consensus_queue.cc:1035] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:06.132761 22375 consensus_queue.cc:1035] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:06.136080 22375 raft_consensus.cc:2932] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:06.136742 16153 raft_consensus.cc:2932] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:06.144021 12832 catalog_manager.cc:5484] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:06.161150 16153 raft_consensus.cc:1203] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:13:06.161013 12974 raft_consensus.cc:2932] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:06.201012 15754 consensus_queue.cc:237] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:06.203697 16023 raft_consensus.cc:1261] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:06.203938 13105 raft_consensus.cc:1261] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:06.204334 22557 consensus_queue.cc:1035] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:06.204699 23242 consensus_queue.cc:1035] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:06.205415 15692 consensus_peers.cc:485] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 181a5dd80e5a448b9a63d037e8ae93e7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:06.205574 23120 ts_tablet_manager.cc:902] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:13:06.229555 22557 raft_consensus.cc:2932] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:06.231858 23120 tablet_copy_client.cc:308] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:13:06.232760 13254 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet 08106d7d42674f01bbee94b6236fb9f4 from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:60419)
I20240501 08:13:06.232942 13254 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet 08106d7d42674f01bbee94b6236fb9f4 from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:60419: session id = 76d830e199eb461c8ca689c9460ee0c2-08106d7d42674f01bbee94b6236fb9f4
I20240501 08:13:06.234366 16023 raft_consensus.cc:2932] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:06.235682 13254 tablet_copy_source_session.cc:205] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:06.238842 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08106d7d42674f01bbee94b6236fb9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:06.245559 23120 tablet_copy_client.cc:783] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:06.244269 12832 catalog_manager.cc:5484] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:06.245780 23120 tablet_copy_client.cc:647] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:06.255232 13105 raft_consensus.cc:2932] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:06.270660 23120 tablet_copy_client.cc:523] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:06.274129 23120 tablet_bootstrap.cc:492] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:06.282681 23120 tablet_bootstrap.cc:492] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:06.283174 23120 tablet_bootstrap.cc:492] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:06.283474 23120 ts_tablet_manager.cc:1367] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240501 08:13:06.284914 23120 raft_consensus.cc:348] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:06.285183 23120 raft_consensus.cc:729] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:06.285401 23120 consensus_queue.cc:260] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:06.285951 23120 ts_tablet_manager.cc:1398] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:13:06.286708 13254 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-08106d7d42674f01bbee94b6236fb9f4 received from {username='slave'} at 127.6.56.65:60419
I20240501 08:13:06.286834 13254 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-08106d7d42674f01bbee94b6236fb9f4 on tablet 08106d7d42674f01bbee94b6236fb9f4 with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:13:06.293797 23173 ts_tablet_manager.cc:902] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:13:06.328955 23173 tablet_copy_client.cc:308] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:13:06.329896 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet 34a5baf3631b45d68a2003fc4539f799 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:60299)
I20240501 08:13:06.330049 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet 34a5baf3631b45d68a2003fc4539f799 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:60299: session id = 4ecb724cb4764a53988fde00407db4c0-34a5baf3631b45d68a2003fc4539f799
I20240501 08:13:06.333024 12994 tablet_copy_source_session.cc:205] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:06.335461 23173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34a5baf3631b45d68a2003fc4539f799. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:06.341913 23173 tablet_copy_client.cc:783] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:06.342139 23173 tablet_copy_client.cc:647] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:06.344233 23173 tablet_copy_client.cc:523] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:06.350919 23173 tablet_bootstrap.cc:492] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:06.354945 15892 consensus_queue.cc:237] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:06.358858 16023 raft_consensus.cc:1261] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:06.358996 15754 raft_consensus.cc:1261] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240501 08:13:06.359467 15825 consensus_peers.cc:485] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66af91fdcce84b779cf6423562b582b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:06.359944 23094 consensus_queue.cc:1035] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:06.360352 23311 consensus_queue.cc:1035] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:06.364373 23173 tablet_bootstrap.cc:492] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:06.364794 23173 tablet_bootstrap.cc:492] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:06.365101 23173 ts_tablet_manager.cc:1367] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240501 08:13:06.365479 22335 raft_consensus.cc:2932] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index -1 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:06.365967 23173 raft_consensus.cc:348] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:06.366180 23173 raft_consensus.cc:729] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:06.366408 23173 consensus_queue.cc:260] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:06.366919 23173 ts_tablet_manager.cc:1398] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:06.367770 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-34a5baf3631b45d68a2003fc4539f799 received from {username='slave'} at 127.6.56.68:60299
I20240501 08:13:06.367888 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-34a5baf3631b45d68a2003fc4539f799 on tablet 34a5baf3631b45d68a2003fc4539f799 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:06.371412 16023 raft_consensus.cc:2932] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index -1 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:06.375669 15754 raft_consensus.cc:2932] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index -1 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:06.380651 12824 catalog_manager.cc:5484] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index -1 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:06.397811 16023 raft_consensus.cc:1203] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:06.413220 12974 raft_consensus.cc:1203] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:06.480420 13234 raft_consensus.cc:1203] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:06.544450 23345 ts_tablet_manager.cc:902] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:06.587632 23120 ts_tablet_manager.cc:902] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:13:06.593623 23345 tablet_copy_client.cc:308] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:06.594436 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet f22fc6797cb24451b78a5c1e5f6306ba from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:58107)
I20240501 08:13:06.594597 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet f22fc6797cb24451b78a5c1e5f6306ba from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:58107: session id = 6ed448d04e4646658c0de37d829ff15b-f22fc6797cb24451b78a5c1e5f6306ba
I20240501 08:13:06.599411 13385 tablet_copy_source_session.cc:205] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:06.600888 23345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f22fc6797cb24451b78a5c1e5f6306ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:06.611909 23345 tablet_copy_client.cc:783] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:06.612342 23345 tablet_copy_client.cc:647] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:06.614135 23345 tablet_copy_client.cc:523] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:06.621904 23345 tablet_bootstrap.cc:492] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:13:06.629917 23345 tablet_bootstrap.cc:492] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:06.630364 23345 tablet_bootstrap.cc:492] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:13:06.630616 23345 ts_tablet_manager.cc:1367] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240501 08:13:06.631333 23345 raft_consensus.cc:348] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:06.631508 23345 raft_consensus.cc:729] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:13:06.631744 23345 consensus_queue.cc:260] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:06.634819 23345 ts_tablet_manager.cc:1398] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:06.637697 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-f22fc6797cb24451b78a5c1e5f6306ba received from {username='slave'} at 127.6.56.67:58107
I20240501 08:13:06.637820 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-f22fc6797cb24451b78a5c1e5f6306ba on tablet f22fc6797cb24451b78a5c1e5f6306ba with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:13:06.657505 23120 tablet_copy_client.cc:308] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:13:06.658479 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet 181a5dd80e5a448b9a63d037e8ae93e7 from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:53631)
I20240501 08:13:06.658656 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet 181a5dd80e5a448b9a63d037e8ae93e7 from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:53631: session id = 76d830e199eb461c8ca689c9460ee0c2-181a5dd80e5a448b9a63d037e8ae93e7
I20240501 08:13:06.673569 15776 tablet_copy_source_session.cc:205] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:06.673840 23366 ts_tablet_manager.cc:902] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:13:06.675093 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 181a5dd80e5a448b9a63d037e8ae93e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:06.678856 23366 tablet_copy_client.cc:308] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:13:06.679708 15775 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet ba08236c71df4c8190f6b7644ed944ed from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:44937)
I20240501 08:13:06.679936 15775 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet ba08236c71df4c8190f6b7644ed944ed from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:44937: session id = 4d516916eeac468db514eeb928f06190-ba08236c71df4c8190f6b7644ed944ed
I20240501 08:13:06.683511 15775 tablet_copy_source_session.cc:205] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:06.685024 23366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba08236c71df4c8190f6b7644ed944ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:06.694977 23120 tablet_copy_client.cc:783] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:06.695163 23120 tablet_copy_client.cc:647] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:06.753428 23366 tablet_copy_client.cc:783] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:06.753990 23366 tablet_copy_client.cc:647] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:06.774147 23366 tablet_copy_client.cc:523] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:06.779299 23366 tablet_bootstrap.cc:492] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:13:06.811908 23120 tablet_copy_client.cc:523] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:06.815578 23120 tablet_bootstrap.cc:492] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:06.830289 12974 raft_consensus.cc:1203] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:06.834131 23120 tablet_bootstrap.cc:492] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:06.834816 23120 tablet_bootstrap.cc:492] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:06.835073 23120 ts_tablet_manager.cc:1367] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.001s
I20240501 08:13:06.835842 23120 raft_consensus.cc:348] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:06.836079 23120 raft_consensus.cc:729] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:06.836357 23120 consensus_queue.cc:260] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:06.837034 23120 ts_tablet_manager.cc:1398] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:06.838572 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-181a5dd80e5a448b9a63d037e8ae93e7 received from {username='slave'} at 127.6.56.65:53631
I20240501 08:13:06.838830 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-181a5dd80e5a448b9a63d037e8ae93e7 on tablet 181a5dd80e5a448b9a63d037e8ae93e7 with peer 76d830e199eb461c8ca689c9460ee0c2
W20240501 08:13:06.860534 13301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 155028us
I20240501 08:13:06.862859 23366 tablet_bootstrap.cc:492] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:06.863337 23366 tablet_bootstrap.cc:492] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:13:06.863590 23366 ts_tablet_manager.cc:1367] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.084s user 0.010s sys 0.000s
I20240501 08:13:06.864478 23366 raft_consensus.cc:348] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:06.866675 23366 raft_consensus.cc:729] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:13:06.866998 23366 consensus_queue.cc:260] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:06.867729 23366 ts_tablet_manager.cc:1398] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:13:06.869150 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-ba08236c71df4c8190f6b7644ed944ed received from {username='slave'} at 127.6.56.70:44937
I20240501 08:13:06.869271 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session 4d516916eeac468db514eeb928f06190-ba08236c71df4c8190f6b7644ed944ed on tablet ba08236c71df4c8190f6b7644ed944ed with peer 4d516916eeac468db514eeb928f06190
W20240501 08:13:06.875285 13041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 43808.6us
I20240501 08:13:06.895382 13365 raft_consensus.cc:1203] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:06.988515 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:06.989027 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:07.001507 23397 ts_tablet_manager.cc:902] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:13:07.046120 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:07.076423 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:07.080931 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:07.107553 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:07.109155 15891 raft_consensus.cc:1203] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:07.144793 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:07.158581 12974 raft_consensus.cc:1203] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:07.162407 13234 raft_consensus.cc:1203] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:07.180825 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:07.315565 23397 tablet_copy_client.cc:308] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:13:07.316363 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 66af91fdcce84b779cf6423562b582b0 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:37921)
I20240501 08:13:07.316543 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 66af91fdcce84b779cf6423562b582b0 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:37921: session id = 4ecb724cb4764a53988fde00407db4c0-66af91fdcce84b779cf6423562b582b0
I20240501 08:13:07.319479 15913 tablet_copy_source_session.cc:205] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:07.320690 23397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66af91fdcce84b779cf6423562b582b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:07.328276 23397 tablet_copy_client.cc:783] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:07.328524 23397 tablet_copy_client.cc:647] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:07.338214 23397 tablet_copy_client.cc:523] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:07.341555 23397 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:07.360275 23397 tablet.cc:1722] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:13:07.360862 23397 tablet.cc:1620] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: MemRowSet was empty: no flush needed.
I20240501 08:13:07.360976 23397 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20240501 08:13:07.362553 23397 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:07.363046 23397 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:07.369750 23397 ts_tablet_manager.cc:1367] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.006s
I20240501 08:13:07.370358 23397 raft_consensus.cc:348] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:07.370532 23397 raft_consensus.cc:729] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:07.370800 23397 consensus_queue.cc:260] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:07.371645 23397 ts_tablet_manager.cc:1398] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:13:07.372558 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-66af91fdcce84b779cf6423562b582b0 received from {username='slave'} at 127.6.56.68:37921
I20240501 08:13:07.372711 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-66af91fdcce84b779cf6423562b582b0 on tablet 66af91fdcce84b779cf6423562b582b0 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:07.569846 13365 raft_consensus.cc:1203] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240501 08:13:07.723291 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:07.726334 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:07.726552 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:07.750926 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:07.777827 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:07.792364 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:07.802539 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:07.830621 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:08.007331 12973 consensus_queue.cc:237] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:08.009493 15892 raft_consensus.cc:1261] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:08.009533 13365 raft_consensus.cc:1261] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:08.009999 23287 consensus_queue.cc:1035] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:08.010325 22897 consensus_queue.cc:1035] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:08.013250 23287 raft_consensus.cc:2932] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:08.014362 13365 raft_consensus.cc:2932] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:08.014314 15892 raft_consensus.cc:2932] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:08.020182 12832 catalog_manager.cc:5484] T 34a5baf3631b45d68a2003fc4539f799 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:08.025157 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 34a5baf3631b45d68a2003fc4539f799 with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40166
I20240501 08:13:08.025795 23496 tablet_replica.cc:330] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:13:08.026034 23496 raft_consensus.cc:2229] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:08.026279 23496 raft_consensus.cc:2258] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:08.026966 23496 ts_tablet_manager.cc:1875] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:08.034588 23496 ts_tablet_manager.cc:1888] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:08.034736 23496 log.cc:1196] T 34a5baf3631b45d68a2003fc4539f799 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/34a5baf3631b45d68a2003fc4539f799
I20240501 08:13:08.035578 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 34a5baf3631b45d68a2003fc4539f799 (table rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]) successfully deleted
I20240501 08:13:08.361804 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:08.417013 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:08.427587 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:08.456771 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:08.530799 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:08.535514 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:08.565668 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:08.598373 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:08.744259 13234 consensus_queue.cc:237] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:08.746637 12973 raft_consensus.cc:1261] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:08.746642 13365 raft_consensus.cc:1261] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:08.747181 22973 consensus_queue.cc:1035] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:08.747769 22973 consensus_queue.cc:1035] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:08.749729 22819 raft_consensus.cc:2932] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:08.750291 12973 raft_consensus.cc:2932] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:08.752508 13365 raft_consensus.cc:2932] T 08106d7d42674f01bbee94b6236fb9f4 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:08.758486 12828 catalog_manager.cc:5484] T 08106d7d42674f01bbee94b6236fb9f4 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:08.771054 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 08106d7d42674f01bbee94b6236fb9f4 with delete_type TABLET_DATA_TOMBSTONED (TS e3fefd4dca2d41678187047f06c10187 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58510
I20240501 08:13:08.776646 23590 tablet_replica.cc:330] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:13:08.776940 23590 raft_consensus.cc:2229] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:08.777186 23590 raft_consensus.cc:2258] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:08.777863 23590 ts_tablet_manager.cc:1875] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:08.783238 23590 ts_tablet_manager.cc:1888] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:08.783347 23590 log.cc:1196] T 08106d7d42674f01bbee94b6236fb9f4 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/08106d7d42674f01bbee94b6236fb9f4
I20240501 08:13:08.784080 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 08106d7d42674f01bbee94b6236fb9f4 (table rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]) successfully deleted
I20240501 08:13:09.163654 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:09.205138 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:09.221817 16138 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:09.224644 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:09.251597 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:09.320410 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:09.335881 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:09.339099 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:09.452404 16153 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8cc25959b69a41c98d196ed88117b26f"
dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40234
I20240501 08:13:09.452572 16153 raft_consensus.cc:595] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:09.684957 22876 raft_consensus.cc:979] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: : Instructing follower e3fefd4dca2d41678187047f06c10187 to start an election
I20240501 08:13:09.685122 22876 raft_consensus.cc:1067] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Signalling peer e3fefd4dca2d41678187047f06c10187 to start an election
I20240501 08:13:09.686915 13104 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8cc25959b69a41c98d196ed88117b26f"
dest_uuid: "e3fefd4dca2d41678187047f06c10187"
from {username='slave'} at 127.6.56.72:42629
I20240501 08:13:09.687105 13104 raft_consensus.cc:482] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:09.687197 13104 raft_consensus.cc:3037] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:09.690065 13104 raft_consensus.cc:504] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:09.693011 16153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc25959b69a41c98d196ed88117b26f" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:13:09.693192 16153 raft_consensus.cc:3032] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:09.693257 16153 raft_consensus.cc:729] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:13:09.693595 16153 consensus_queue.cc:260] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:09.693985 16153 raft_consensus.cc:3037] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:09.699687 12974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc25959b69a41c98d196ed88117b26f" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:13:09.699903 12974 raft_consensus.cc:3037] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:09.701493 13104 leader_election.cc:288] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:13:09.701529 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc25959b69a41c98d196ed88117b26f" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:13:09.701732 13232 raft_consensus.cc:3037] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:09.702626 16153 raft_consensus.cc:2454] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 2.
I20240501 08:13:09.704154 13232 raft_consensus.cc:2454] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 2.
I20240501 08:13:09.705672 12974 raft_consensus.cc:2454] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 2.
I20240501 08:13:09.712637 13041 leader_election.cc:302] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:13:09.713186 23380 raft_consensus.cc:2781] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:09.713343 23380 raft_consensus.cc:686] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 2 LEADER]: Becoming Leader. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:13:09.713624 23380 consensus_queue.cc:237] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:09.724442 12825 catalog_manager.cc:5484] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 reported cstate change: term changed from 1 to 2, leader changed from 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) to e3fefd4dca2d41678187047f06c10187 (127.6.56.66). New cstate: current_term: 2 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:09.796846 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:09.828137 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:09.828718 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:09.829532 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:09.839510 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:09.851008 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:09.892664 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:09.901520 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:10.197976 12974 raft_consensus.cc:1261] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:10.198580 23380 consensus_queue.cc:1035] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:10.208266 13232 raft_consensus.cc:1261] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:10.208992 23380 consensus_queue.cc:1035] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:10.261580 16157 raft_consensus.cc:1261] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:10.262713 23380 consensus_queue.cc:1035] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:10.344962 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:10.373000 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:10.392663 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:10.420504 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:10.451627 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:10.455386 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:10.474746 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:10.534145 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:10.683547 16157 consensus_queue.cc:237] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:10.685395 16022 raft_consensus.cc:1261] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:10.685424 15753 raft_consensus.cc:1261] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:10.685914 23472 consensus_queue.cc:1035] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:10.686540 22876 consensus_queue.cc:1035] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:10.689141 23472 raft_consensus.cc:2932] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:10.689836 15753 raft_consensus.cc:2932] T cabfd31e8ee941cf9f69bc04a0510d11 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:10.701997 12827 catalog_manager.cc:5484] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:10.705648 16022 raft_consensus.cc:2932] T cabfd31e8ee941cf9f69bc04a0510d11 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:10.727681 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet cabfd31e8ee941cf9f69bc04a0510d11 with delete_type TABLET_DATA_TOMBSTONED (TS 4d516916eeac468db514eeb928f06190 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44232
I20240501 08:13:10.728854 23830 tablet_replica.cc:330] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:13:10.729537 23830 raft_consensus.cc:2229] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:10.729846 23830 raft_consensus.cc:2258] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:10.730875 23830 ts_tablet_manager.cc:1875] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:10.737265 23830 ts_tablet_manager.cc:1888] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:10.737469 23830 log.cc:1196] T cabfd31e8ee941cf9f69bc04a0510d11 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/cabfd31e8ee941cf9f69bc04a0510d11
I20240501 08:13:10.738268 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet cabfd31e8ee941cf9f69bc04a0510d11 (table rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]) successfully deleted
I20240501 08:13:10.994491 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:11.025128 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:11.058306 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:11.068573 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:11.080699 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:11.192021 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:11.216902 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:11.260984 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:11.375550 15892 consensus_queue.cc:237] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:11.377499 16153 raft_consensus.cc:1261] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:11.377573 13363 raft_consensus.cc:1261] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:11.378021 23094 consensus_queue.cc:1035] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:11.378307 23632 consensus_queue.cc:1035] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:11.381114 23094 raft_consensus.cc:2932] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:11.382213 13363 raft_consensus.cc:2932] T 85056837a567420a8b480b34b6017b40 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:11.383949 16153 raft_consensus.cc:2932] T 85056837a567420a8b480b34b6017b40 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:11.389209 12831 catalog_manager.cc:5484] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:11.400147 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 85056837a567420a8b480b34b6017b40 with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40166
I20240501 08:13:11.401211 23913 tablet_replica.cc:330] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:13:11.401419 23913 raft_consensus.cc:2229] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:11.401633 23913 raft_consensus.cc:2258] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:11.402240 23913 ts_tablet_manager.cc:1875] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:11.408586 23913 ts_tablet_manager.cc:1888] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:11.408771 23913 log.cc:1196] T 85056837a567420a8b480b34b6017b40 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/85056837a567420a8b480b34b6017b40
I20240501 08:13:11.409976 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 85056837a567420a8b480b34b6017b40 (table rebalancer_extra_table_2 [id=970c6326810a49e99e77ee988599335e]) successfully deleted
I20240501 08:13:11.691602 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:11.704798 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:11.729998 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:11.778638 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:11.839322 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:11.842295 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:11.860530 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:11.868541 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:12.057037 16024 consensus_queue.cc:237] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:12.059217 12972 raft_consensus.cc:1261] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:12.059788 15892 raft_consensus.cc:1261] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:12.059819 23977 consensus_queue.cc:1035] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:12.061187 23977 consensus_queue.cc:1035] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:12.063880 23827 raft_consensus.cc:2932] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:12.064841 15892 raft_consensus.cc:2932] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:12.067965 12972 raft_consensus.cc:2932] T 5e61276f6f4c40f8987747b997ac52ac P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:12.072921 12832 catalog_manager.cc:5484] T 5e61276f6f4c40f8987747b997ac52ac P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:12.095520 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e61276f6f4c40f8987747b997ac52ac with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34524
I20240501 08:13:12.097141 23995 tablet_replica.cc:330] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:13:12.098625 23995 raft_consensus.cc:2229] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:12.098888 23995 raft_consensus.cc:2258] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:12.100031 23995 ts_tablet_manager.cc:1875] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:12.106287 23995 ts_tablet_manager.cc:1888] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:12.106503 23995 log.cc:1196] T 5e61276f6f4c40f8987747b997ac52ac P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/5e61276f6f4c40f8987747b997ac52ac
I20240501 08:13:12.107543 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 5e61276f6f4c40f8987747b997ac52ac (table rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]) successfully deleted
I20240501 08:13:12.346825 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:12.371755 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:12.392454 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:12.457294 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:12.506443 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:12.547192 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:12.555298 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:12.592799 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:12.698115 13104 consensus_queue.cc:237] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:12.700100 16154 raft_consensus.cc:1261] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:12.700184 13234 raft_consensus.cc:1261] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:12.700726 23462 consensus_queue.cc:1035] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:12.701032 23944 consensus_queue.cc:1035] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:12.703774 23462 raft_consensus.cc:2932] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:12.704416 13234 raft_consensus.cc:2932] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:12.704731 16154 raft_consensus.cc:2932] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:12.716879 12829 catalog_manager.cc:5484] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:12.729030 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:13:12.730722 24074 tablet_replica.cc:330] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:13:12.730931 24074 raft_consensus.cc:2229] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:12.731309 24074 raft_consensus.cc:2258] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:12.732178 24074 ts_tablet_manager.cc:1875] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:12.740620 24074 ts_tablet_manager.cc:1888] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:13:12.740772 24074 log.cc:1196] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/99ecb2ec0b3d4fd7a02896cc9a28ba27
I20240501 08:13:12.741698 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 (table rebalance_test_table_1 [id=6c085de9eb3e44d3a1100a754cacad29]) successfully deleted
I20240501 08:13:13.060969 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:13.077941 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:13.080354 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:13.084582 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:13.092022 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:13.139334 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:13.149720 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:13.202961 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:13.312345 13363 consensus_queue.cc:237] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:13.314276 12972 raft_consensus.cc:1261] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:13.314793 13232 raft_consensus.cc:1261] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:13.314780 23885 consensus_queue.cc:1035] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:13.315279 23885 consensus_queue.cc:1035] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:13.317610 24141 raft_consensus.cc:2932] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:13.321009 12972 raft_consensus.cc:2932] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:13.326825 12824 catalog_manager.cc:5484] T f22fc6797cb24451b78a5c1e5f6306ba P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:13.335546 13232 raft_consensus.cc:2932] T f22fc6797cb24451b78a5c1e5f6306ba P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:13.348238 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet f22fc6797cb24451b78a5c1e5f6306ba with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:13:13.349120 24169 tablet_replica.cc:330] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:13:13.350976 24169 raft_consensus.cc:2229] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:13.352216 24169 raft_consensus.cc:2258] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:13.353017 24169 ts_tablet_manager.cc:1875] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:13.358480 24169 ts_tablet_manager.cc:1888] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:13.358595 24169 log.cc:1196] T f22fc6797cb24451b78a5c1e5f6306ba P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/f22fc6797cb24451b78a5c1e5f6306ba
I20240501 08:13:13.359457 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet f22fc6797cb24451b78a5c1e5f6306ba (table rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]) successfully deleted
I20240501 08:13:13.596969 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:13.610759 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:13.634490 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:13.695655 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:13.721261 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:13.729795 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:13.736052 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:13.774991 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:13.897679 15755 consensus_queue.cc:237] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:13.900022 15893 raft_consensus.cc:1261] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:13.900053 13232 raft_consensus.cc:1261] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:13.900990 23212 consensus_queue.cc:1035] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:13.901268 23920 consensus_queue.cc:1035] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:13.910122 23212 raft_consensus.cc:2932] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:13.913934 15892 raft_consensus.cc:2932] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:13.926277 12832 catalog_manager.cc:5484] T ba08236c71df4c8190f6b7644ed944ed P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:13.959609 13232 raft_consensus.cc:2932] T ba08236c71df4c8190f6b7644ed944ed P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:13.971182 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet ba08236c71df4c8190f6b7644ed944ed with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:13:13.979068 24254 tablet_replica.cc:330] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:13:13.979312 24254 raft_consensus.cc:2229] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:13.979521 24254 raft_consensus.cc:2258] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:13.981369 24254 ts_tablet_manager.cc:1875] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:13.988063 24254 ts_tablet_manager.cc:1888] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:13.988194 24254 log.cc:1196] T ba08236c71df4c8190f6b7644ed944ed P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/ba08236c71df4c8190f6b7644ed944ed
I20240501 08:13:13.989040 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet ba08236c71df4c8190f6b7644ed944ed (table rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]) successfully deleted
I20240501 08:13:14.249867 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:14.270680 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:14.314007 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:14.343613 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:14.357851 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:14.372498 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:14.388960 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:14.416765 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:14.583034 13105 consensus_queue.cc:237] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:14.595600 15892 raft_consensus.cc:1261] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:14.596521 23944 consensus_queue.cc:1035] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:14.599110 13363 raft_consensus.cc:1261] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:14.605544 23462 consensus_queue.cc:1035] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:14.606228 23462 raft_consensus.cc:2932] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:14.611776 15892 raft_consensus.cc:2932] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:14.615768 13363 raft_consensus.cc:2932] T a412b8ae0aa648d7979fac58b50d86a6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:14.618208 12832 catalog_manager.cc:5484] T a412b8ae0aa648d7979fac58b50d86a6 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:14.649547 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet a412b8ae0aa648d7979fac58b50d86a6 with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57838
I20240501 08:13:14.663054 24345 tablet_replica.cc:330] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:13:14.663262 24345 raft_consensus.cc:2229] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:14.663614 24345 raft_consensus.cc:2258] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:14.664373 24345 ts_tablet_manager.cc:1875] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:14.669802 24345 ts_tablet_manager.cc:1888] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:13:14.669968 24345 log.cc:1196] T a412b8ae0aa648d7979fac58b50d86a6 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/a412b8ae0aa648d7979fac58b50d86a6
I20240501 08:13:14.671164 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet a412b8ae0aa648d7979fac58b50d86a6 (table rebalance_test_table_9 [id=5a04eed287ab4f3ead90a7aab3a14938]) successfully deleted
I20240501 08:13:14.893481 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:14.902235 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:14.908861 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:14.926962 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:15.011368 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:15.012323 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:15.050802 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:15.082052 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:15.212824 15892 consensus_queue.cc:237] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:15.217563 13365 raft_consensus.cc:1261] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:15.218425 16024 raft_consensus.cc:1261] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:15.218495 24216 consensus_queue.cc:1035] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:15.219801 24216 consensus_queue.cc:1035] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:15.223001 24216 raft_consensus.cc:2932] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:15.224961 16024 raft_consensus.cc:2932] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:15.230815 12830 catalog_manager.cc:5484] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 3 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:15.233691 13365 raft_consensus.cc:2932] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:15.256467 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 66af91fdcce84b779cf6423562b582b0 with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40166
I20240501 08:13:15.268739 24428 tablet_replica.cc:330] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:13:15.269008 24428 raft_consensus.cc:2229] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:15.269258 24428 raft_consensus.cc:2258] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:15.269831 24428 ts_tablet_manager.cc:1875] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:15.275389 24428 ts_tablet_manager.cc:1888] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 08:13:15.275494 24428 log.cc:1196] T 66af91fdcce84b779cf6423562b582b0 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/66af91fdcce84b779cf6423562b582b0
I20240501 08:13:15.276180 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 66af91fdcce84b779cf6423562b582b0 (table rebalancer_dynamic_table [id=b6b6af5609ce409ba8be11e257f107ef]) successfully deleted
I20240501 08:13:15.522022 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:15.601114 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:15.612468 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:15.623447 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:15.625883 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:15.636847 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 08:13:15.637796 15691 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 71668.5us
I20240501 08:13:15.655370 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 08:13:15.687765 15691 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.66:35509, user_credentials={real_user=slave}} blocked reactor thread for 46493.5us
I20240501 08:13:15.708629 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:15.819281 13105 consensus_queue.cc:237] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:15.821223 15755 raft_consensus.cc:1261] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:15.821781 15892 raft_consensus.cc:1261] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:15.821784 24334 consensus_queue.cc:1035] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:15.822263 24337 consensus_queue.cc:1035] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:15.824151 24334 raft_consensus.cc:2932] T c99eddd74b7740dcab6b93ca437b51ae P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:15.824759 15755 raft_consensus.cc:2932] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:15.826512 15892 raft_consensus.cc:2932] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:15.832607 12831 catalog_manager.cc:5484] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:15.861392 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet c99eddd74b7740dcab6b93ca437b51ae with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34524
I20240501 08:13:15.864809 24512 tablet_replica.cc:330] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:13:15.866643 24512 raft_consensus.cc:2229] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:15.867218 24512 raft_consensus.cc:2258] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:15.868569 24512 ts_tablet_manager.cc:1875] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:15.874435 24512 ts_tablet_manager.cc:1888] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:15.874554 24512 log.cc:1196] T c99eddd74b7740dcab6b93ca437b51ae P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/c99eddd74b7740dcab6b93ca437b51ae
I20240501 08:13:15.875222 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet c99eddd74b7740dcab6b93ca437b51ae (table rebalancer_extra_table_52 [id=5e85f6f5c8c64e44a9591cefab6182ef]) successfully deleted
I20240501 08:13:16.137532 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:16.137868 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:16.138823 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:16.178086 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:16.179595 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:16.184068 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:16.214267 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:16.247977 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:16.406838 13234 consensus_queue.cc:237] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:16.409162 16154 raft_consensus.cc:1261] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:16.409204 13105 raft_consensus.cc:1261] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:16.409761 24314 consensus_queue.cc:1035] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:16.411629 24314 consensus_queue.cc:1035] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:16.416676 23695 raft_consensus.cc:2932] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:16.417328 16154 raft_consensus.cc:2932] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:16.420284 13104 raft_consensus.cc:2932] T 5741e5458ccd4af6a3972495ccf40b8f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:16.423699 12827 catalog_manager.cc:5484] T 5741e5458ccd4af6a3972495ccf40b8f P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:16.433480 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet 5741e5458ccd4af6a3972495ccf40b8f with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:13:16.434794 24589 tablet_replica.cc:330] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:13:16.435002 24589 raft_consensus.cc:2229] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:16.435227 24589 raft_consensus.cc:2258] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:16.435916 24589 ts_tablet_manager.cc:1875] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:16.441537 24589 ts_tablet_manager.cc:1888] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:13:16.441884 24589 log.cc:1196] T 5741e5458ccd4af6a3972495ccf40b8f P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/5741e5458ccd4af6a3972495ccf40b8f
I20240501 08:13:16.442589 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 5741e5458ccd4af6a3972495ccf40b8f (table rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]) successfully deleted
I20240501 08:13:16.732424 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:16.759052 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:16.760278 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:16.764673 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:16.789943 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:16.821040 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:16.849787 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:16.877009 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:17.070124 13365 consensus_queue.cc:237] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:17.072027 16154 raft_consensus.cc:1261] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:17.072027 12971 raft_consensus.cc:1261] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:17.072564 24399 consensus_queue.cc:1035] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:17.072816 24312 consensus_queue.cc:1035] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:17.075718 24399 raft_consensus.cc:2932] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:17.076087 12971 raft_consensus.cc:2932] T a6154bf2fadb45e58c27c46165b5824f P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:17.076263 16154 raft_consensus.cc:2932] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:17.082554 12832 catalog_manager.cc:5484] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:17.090960 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet a6154bf2fadb45e58c27c46165b5824f with delete_type TABLET_DATA_TOMBSTONED (TS 4d516916eeac468db514eeb928f06190 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44232
I20240501 08:13:17.091517 24688 tablet_replica.cc:330] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:13:17.091722 24688 raft_consensus.cc:2229] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:17.091936 24688 raft_consensus.cc:2258] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:17.092634 24688 ts_tablet_manager.cc:1875] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:17.098518 24688 ts_tablet_manager.cc:1888] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:17.098707 24688 log.cc:1196] T a6154bf2fadb45e58c27c46165b5824f P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/a6154bf2fadb45e58c27c46165b5824f
I20240501 08:13:17.099740 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet a6154bf2fadb45e58c27c46165b5824f (table rebalancer_extra_table_32 [id=be8478913ef642319e7d9cfac923eac4]) successfully deleted
I20240501 08:13:17.586611 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:17.599114 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:17.651827 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:17.668365 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:17.678840 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:17.722532 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:17.739022 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:17.758636 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:17.890151 15893 consensus_queue.cc:237] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:17.892369 15755 raft_consensus.cc:1261] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:17.892383 13103 raft_consensus.cc:1261] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:17.892982 24306 consensus_queue.cc:1035] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:17.894253 24565 consensus_queue.cc:1035] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:17.914964 24306 raft_consensus.cc:2932] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:17.922626 15755 raft_consensus.cc:2932] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:17.932639 12829 catalog_manager.cc:5484] T f8203d209d624d2695e5b32bbfa0edf7 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:17.972273 13103 raft_consensus.cc:2932] T f8203d209d624d2695e5b32bbfa0edf7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
W20240501 08:13:17.981293 15823 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 86670.9us
I20240501 08:13:17.991621 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet f8203d209d624d2695e5b32bbfa0edf7 with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:13:18.000474 24784 tablet_replica.cc:330] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:13:18.000793 24784 raft_consensus.cc:2229] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:18.001116 24784 raft_consensus.cc:2258] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:18.001843 24784 ts_tablet_manager.cc:1875] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:18.012861 24784 ts_tablet_manager.cc:1888] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:18.013088 24784 log.cc:1196] T f8203d209d624d2695e5b32bbfa0edf7 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/f8203d209d624d2695e5b32bbfa0edf7
I20240501 08:13:18.013968 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet f8203d209d624d2695e5b32bbfa0edf7 (table rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]) successfully deleted
I20240501 08:13:18.090516 15893 consensus_queue.cc:237] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:18.094602 16022 raft_consensus.cc:1261] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:18.094916 13363 raft_consensus.cc:1261] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:18.095592 15826 consensus_peers.cc:485] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e330f6ac7074134ba06effe5ececcde. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.095638 24776 consensus_queue.cc:1035] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:18.096005 24779 consensus_queue.cc:1035] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:18.098788 24776 raft_consensus.cc:2932] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:18.104219 16022 raft_consensus.cc:2932] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:18.111995 12827 catalog_manager.cc:5484] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:18.124066 13363 raft_consensus.cc:2932] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:18.161597 13363 consensus_queue.cc:237] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:18.164196 12974 raft_consensus.cc:1261] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:18.164567 13103 raft_consensus.cc:1261] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:18.164983 24679 consensus_queue.cc:1035] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:13:18.165407 13298 consensus_peers.cc:485] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74d1a1cd0bf94d0d9a87674d41838d2d. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.165656 24312 consensus_queue.cc:1035] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:18.173504 24679 raft_consensus.cc:2932] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:18.174126 13104 raft_consensus.cc:2932] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:18.176991 12974 raft_consensus.cc:2932] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:18.179220 12825 catalog_manager.cc:5484] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:18.216526 15892 consensus_queue.cc:237] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:18.219046 13363 raft_consensus.cc:1261] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:18.219245 15825 consensus_peers.cc:485] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f305a6e24ab1436b895db037f5d72870. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:18.219594 24776 consensus_queue.cc:1035] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:18.219908 16154 raft_consensus.cc:1261] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:18.220585 24776 consensus_queue.cc:1035] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:18.233559 24776 raft_consensus.cc:2932] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:18.234194 13363 raft_consensus.cc:2932] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:18.241196 16157 raft_consensus.cc:2932] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:18.241343 12825 catalog_manager.cc:5484] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:18.520617 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:18.547937 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:18.554126 24907 ts_tablet_manager.cc:902] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:13:18.558827 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:18.562053 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:18.568270 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:18.657136 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:18.678524 24914 ts_tablet_manager.cc:902] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:18.696352 24907 tablet_copy_client.cc:308] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:13:18.697072 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 4e330f6ac7074134ba06effe5ececcde from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:42849)
I20240501 08:13:18.697229 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 4e330f6ac7074134ba06effe5ececcde from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:42849: session id = 6ed448d04e4646658c0de37d829ff15b-4e330f6ac7074134ba06effe5ececcde
I20240501 08:13:18.697309 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:18.737118 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:18.741613 24922 ts_tablet_manager.cc:902] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:13:18.755476 15913 tablet_copy_source_session.cc:205] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:18.760706 24907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e330f6ac7074134ba06effe5ececcde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:18.767516 24907 tablet_copy_client.cc:783] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:18.767890 24907 tablet_copy_client.cc:647] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:18.769593 24907 tablet_copy_client.cc:523] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:18.772502 24914 tablet_copy_client.cc:308] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:18.773335 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 74d1a1cd0bf94d0d9a87674d41838d2d from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:45907)
I20240501 08:13:18.773543 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 74d1a1cd0bf94d0d9a87674d41838d2d from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:45907: session id = 4d516916eeac468db514eeb928f06190-74d1a1cd0bf94d0d9a87674d41838d2d
I20240501 08:13:18.776338 24922 tablet_copy_client.cc:308] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:13:18.777020 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet f305a6e24ab1436b895db037f5d72870 from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:48647)
I20240501 08:13:18.777200 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet f305a6e24ab1436b895db037f5d72870 from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:48647: session id = 982f49fc87f149439cda0de2261b6bfd-f305a6e24ab1436b895db037f5d72870
I20240501 08:13:18.777035 24907 tablet_bootstrap.cc:492] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:13:18.780268 15913 tablet_copy_source_session.cc:205] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:18.781667 24922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f305a6e24ab1436b895db037f5d72870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:18.780220 13385 tablet_copy_source_session.cc:205] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:18.789031 24922 tablet_copy_client.cc:783] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:18.789309 24922 tablet_copy_client.cc:647] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:18.790946 24922 tablet_copy_client.cc:523] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:18.795074 24922 tablet_bootstrap.cc:492] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:13:18.795917 24914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74d1a1cd0bf94d0d9a87674d41838d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:18.808161 24922 tablet_bootstrap.cc:492] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:18.808661 24922 tablet_bootstrap.cc:492] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:13:18.809084 24922 ts_tablet_manager.cc:1367] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.008s
I20240501 08:13:18.809914 24922 raft_consensus.cc:348] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:18.810276 24922 raft_consensus.cc:729] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:13:18.810570 24922 consensus_queue.cc:260] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:18.811288 24922 ts_tablet_manager.cc:1398] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:13:18.812181 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-f305a6e24ab1436b895db037f5d72870 received from {username='slave'} at 127.6.56.69:48647
I20240501 08:13:18.812322 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-f305a6e24ab1436b895db037f5d72870 on tablet f305a6e24ab1436b895db037f5d72870 with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:13:18.818521 24914 tablet_copy_client.cc:783] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:18.818794 24914 tablet_copy_client.cc:647] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:18.821625 24914 tablet_copy_client.cc:523] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:18.826200 24914 tablet_bootstrap.cc:492] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:13:18.852309 24907 tablet_bootstrap.cc:492] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:18.853031 24907 tablet_bootstrap.cc:492] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:13:18.853364 24907 ts_tablet_manager.cc:1367] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.013s
I20240501 08:13:18.854101 24907 raft_consensus.cc:348] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:18.854329 24907 raft_consensus.cc:729] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:13:18.854616 24907 consensus_queue.cc:260] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:18.855361 24907 ts_tablet_manager.cc:1398] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:18.856812 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-4e330f6ac7074134ba06effe5ececcde received from {username='slave'} at 127.6.56.67:42849
I20240501 08:13:18.856930 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-4e330f6ac7074134ba06effe5ececcde on tablet 4e330f6ac7074134ba06effe5ececcde with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:13:18.882606 24914 tablet_bootstrap.cc:492] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:18.882988 24914 tablet_bootstrap.cc:492] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:13:18.883229 24914 ts_tablet_manager.cc:1367] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.057s user 0.043s sys 0.006s
I20240501 08:13:18.883773 24914 raft_consensus.cc:348] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:18.883932 24914 raft_consensus.cc:729] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:13:18.884119 24914 consensus_queue.cc:260] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:18.884581 24914 ts_tablet_manager.cc:1398] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 08:13:18.885257 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-74d1a1cd0bf94d0d9a87674d41838d2d received from {username='slave'} at 127.6.56.70:45907
I20240501 08:13:18.885360 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 4d516916eeac468db514eeb928f06190-74d1a1cd0bf94d0d9a87674d41838d2d on tablet 74d1a1cd0bf94d0d9a87674d41838d2d with peer 4d516916eeac468db514eeb928f06190
I20240501 08:13:18.946689 13234 raft_consensus.cc:1203] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:19.114468 15892 raft_consensus.cc:1203] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:13:19.164534 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:19.166839 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:19.197491 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:19.247845 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:19.256742 15754 raft_consensus.cc:1203] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:19.314770 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:19.332537 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:19.373816 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:19.400211 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:19.512660 13365 consensus_queue.cc:237] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:19.521911 12972 raft_consensus.cc:1261] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:19.523782 24312 consensus_queue.cc:1035] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:19.529136 15892 raft_consensus.cc:1261] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:19.530301 24312 consensus_queue.cc:1035] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:19.532862 24312 raft_consensus.cc:2932] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:19.536574 15892 raft_consensus.cc:2932] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:19.541157 12824 catalog_manager.cc:5484] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:19.601917 12974 raft_consensus.cc:2932] T 74d1a1cd0bf94d0d9a87674d41838d2d P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:19.631511 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 74d1a1cd0bf94d0d9a87674d41838d2d with delete_type TABLET_DATA_TOMBSTONED (TS e3fefd4dca2d41678187047f06c10187 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:58510
I20240501 08:13:19.636716 25025 tablet_replica.cc:330] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:13:19.637238 25025 raft_consensus.cc:2229] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:19.637599 25025 raft_consensus.cc:2258] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:19.641561 25025 ts_tablet_manager.cc:1875] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:19.647567 25025 ts_tablet_manager.cc:1888] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:13:19.647675 25025 log.cc:1196] T 74d1a1cd0bf94d0d9a87674d41838d2d P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/74d1a1cd0bf94d0d9a87674d41838d2d
I20240501 08:13:19.648412 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 74d1a1cd0bf94d0d9a87674d41838d2d (table rebalance_test_table_17 [id=aa780586626d49fe8f25ab6348b6cdac]) successfully deleted
I20240501 08:13:19.860229 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:19.872759 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:19.920737 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:19.925504 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:19.953938 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:19.976459 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:20.000872 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:20.055549 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:20.195111 15892 consensus_queue.cc:237] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:20.197070 13365 raft_consensus.cc:1261] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:20.197057 13235 raft_consensus.cc:1261] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:20.197558 24776 consensus_queue.cc:1035] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:20.197834 24565 consensus_queue.cc:1035] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:20.200886 24776 raft_consensus.cc:2932] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:20.201686 13235 raft_consensus.cc:2932] T 4e330f6ac7074134ba06effe5ececcde P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:20.202491 13365 raft_consensus.cc:2932] T 4e330f6ac7074134ba06effe5ececcde P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:20.207134 12829 catalog_manager.cc:5484] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:20.215957 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e330f6ac7074134ba06effe5ececcde with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:13:20.216607 25102 tablet_replica.cc:330] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:13:20.216888 25102 raft_consensus.cc:2229] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:20.217175 25102 raft_consensus.cc:2258] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:20.217898 25102 ts_tablet_manager.cc:1875] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:20.224486 25102 ts_tablet_manager.cc:1888] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:20.224627 25102 log.cc:1196] T 4e330f6ac7074134ba06effe5ececcde P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/4e330f6ac7074134ba06effe5ececcde
I20240501 08:13:20.225540 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 4e330f6ac7074134ba06effe5ececcde (table rebalancer_extra_table_14 [id=ab42c80aeb0a45eaa2671b7f02c67e6d]) successfully deleted
I20240501 08:13:20.471719 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:20.507511 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:20.532524 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:20.539988 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:20.597486 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:20.613647 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:20.645473 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:20.782505 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:20.877497 15754 consensus_queue.cc:237] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:20.879658 12972 raft_consensus.cc:1261] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:20.879973 16024 raft_consensus.cc:1261] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:20.880228 24718 consensus_queue.cc:1035] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:20.880513 25031 consensus_queue.cc:1035] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:20.885587 25031 raft_consensus.cc:2932] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:20.888900 12972 raft_consensus.cc:2932] T 181a5dd80e5a448b9a63d037e8ae93e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:20.897388 12829 catalog_manager.cc:5484] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:20.900800 16024 raft_consensus.cc:2932] T 181a5dd80e5a448b9a63d037e8ae93e7 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:20.930861 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 181a5dd80e5a448b9a63d037e8ae93e7 with delete_type TABLET_DATA_TOMBSTONED (TS e3fefd4dca2d41678187047f06c10187 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:58510
I20240501 08:13:20.945222 25188 tablet_replica.cc:330] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:13:20.945720 25188 raft_consensus.cc:2229] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:20.946125 25188 raft_consensus.cc:2258] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:20.947198 25188 ts_tablet_manager.cc:1875] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:20.956702 25188 ts_tablet_manager.cc:1888] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:20.957010 25188 log.cc:1196] T 181a5dd80e5a448b9a63d037e8ae93e7 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/181a5dd80e5a448b9a63d037e8ae93e7
I20240501 08:13:20.958094 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 181a5dd80e5a448b9a63d037e8ae93e7 (table rebalancer_extra_table_8 [id=cafe24b0c9be43228f232abd6b744fca]) successfully deleted
I20240501 08:13:21.270756 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:21.301986 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:21.334972 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:21.346210 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:21.347083 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:21.362856 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:21.418473 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:21.424988 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:21.584270 13103 consensus_queue.cc:237] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:21.593631 12972 raft_consensus.cc:1261] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:21.593695 13234 raft_consensus.cc:1261] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:21.594240 25110 consensus_queue.cc:1035] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:21.594764 25110 consensus_queue.cc:1035] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:21.597440 25256 raft_consensus.cc:2932] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:21.603348 13234 raft_consensus.cc:2932] T 8cc25959b69a41c98d196ed88117b26f P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:21.603533 12833 catalog_manager.cc:5484] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 2 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:21.607100 12972 raft_consensus.cc:2932] T 8cc25959b69a41c98d196ed88117b26f P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:21.619206 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc25959b69a41c98d196ed88117b26f with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50114
I20240501 08:13:21.623313 25283 tablet_replica.cc:330] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:13:21.623556 25283 raft_consensus.cc:2229] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:21.623731 25283 raft_consensus.cc:2258] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:21.624394 25283 ts_tablet_manager.cc:1875] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:21.630111 25283 ts_tablet_manager.cc:1888] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 08:13:21.630239 25283 log.cc:1196] T 8cc25959b69a41c98d196ed88117b26f P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/8cc25959b69a41c98d196ed88117b26f
I20240501 08:13:21.631177 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 8cc25959b69a41c98d196ed88117b26f (table rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]) successfully deleted
I20240501 08:13:21.764859 13234 consensus_queue.cc:237] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:21.767719 13103 raft_consensus.cc:1261] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:21.768049 13363 raft_consensus.cc:1261] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:13:21.768425 13170 consensus_peers.cc:485] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6fb95e3405ae45d3bfe71fb9d552e3fe. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:21.768733 24314 consensus_queue.cc:1035] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:21.769037 25355 consensus_queue.cc:1035] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:21.773409 25355 raft_consensus.cc:2932] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:21.773972 13363 raft_consensus.cc:2932] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:21.779162 13104 raft_consensus.cc:2932] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:21.780755 12830 catalog_manager.cc:5484] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:21.822980 16024 consensus_queue.cc:237] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:21.825994 12972 raft_consensus.cc:1261] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:21.825976 15753 raft_consensus.cc:1261] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:21.826731 15957 consensus_peers.cc:485] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2e7c8c1795d4c6d997c75e63dee6760. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:21.827139 24566 consensus_queue.cc:1035] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:21.827816 24566 consensus_queue.cc:1035] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:21.831190 25374 raft_consensus.cc:2932] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:21.831746 12972 raft_consensus.cc:2932] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:21.837752 12830 catalog_manager.cc:5484] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 2, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:21.838976 15753 raft_consensus.cc:2932] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:21.911680 16157 consensus_queue.cc:237] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
W20240501 08:13:21.914583 16092 consensus_peers.cc:485] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b000ec8b190644f2883e5f824463b762. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:21.915210 13104 raft_consensus.cc:1261] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:21.915779 25114 consensus_queue.cc:1035] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:21.920475 16024 raft_consensus.cc:1261] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:21.922374 25114 consensus_queue.cc:1035] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:21.938582 16024 raft_consensus.cc:2932] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:21.938061 25396 raft_consensus.cc:2932] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:21.945629 12830 catalog_manager.cc:5484] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:21.948535 13104 raft_consensus.cc:2932] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:22.002323 13363 consensus_queue.cc:237] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:22.005260 12972 raft_consensus.cc:1261] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:13:22.005445 13298 consensus_peers.cc:485] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a98388bf2fd402b8064d2cca06f5b8b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:22.005813 25159 consensus_queue.cc:1035] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:22.006019 13234 raft_consensus.cc:1261] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:22.006484 25159 consensus_queue.cc:1035] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:22.008805 24312 raft_consensus.cc:2932] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:22.009342 13234 raft_consensus.cc:2932] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:22.015393 12824 catalog_manager.cc:5484] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:22.030618 12972 raft_consensus.cc:2932] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:22.077903 13104 consensus_queue.cc:237] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:22.080204 15753 raft_consensus.cc:1261] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:22.080451 13041 consensus_peers.cc:485] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 949d6765e68146d998ff4f5d2b120d68. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:22.080801 15892 raft_consensus.cc:1261] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.080854 25256 consensus_queue.cc:1035] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.081264 25110 consensus_queue.cc:1035] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.084121 25256 raft_consensus.cc:2932] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:22.084756 15753 raft_consensus.cc:2932] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:22.090354 12824 catalog_manager.cc:5484] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:22.091557 15892 raft_consensus.cc:2932] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:22.132074 16157 consensus_queue.cc:237] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:22.134734 13365 raft_consensus.cc:1261] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:22.134747 16092 consensus_peers.cc:485] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1eac68af4bc3473487bf91670cb48223. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:22.135136 15753 raft_consensus.cc:1261] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.135201 25397 consensus_queue.cc:1035] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.135617 25396 consensus_queue.cc:1035] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.137925 25397 raft_consensus.cc:2932] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:22.138446 15753 raft_consensus.cc:2932] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:22.143210 12830 catalog_manager.cc:5484] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:22.144434 13365 raft_consensus.cc:2932] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:22.180233 13365 consensus_queue.cc:237] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:22.182752 15892 raft_consensus.cc:1261] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.182834 16157 raft_consensus.cc:1261] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:22.182900 13301 consensus_peers.cc:485] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65a3d07808414c50a330da007c8f5e19. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:22.183209 24312 consensus_queue.cc:1035] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.183470 25417 consensus_queue.cc:1035] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.186261 24312 raft_consensus.cc:2932] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:22.186903 15892 raft_consensus.cc:2932] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:22.189460 16154 raft_consensus.cc:2932] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:22.192958 12830 catalog_manager.cc:5484] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:22.207669 25479 ts_tablet_manager.cc:902] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:22.219691 25479 tablet_copy_client.cc:308] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:13:22.220650 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet b2e7c8c1795d4c6d997c75e63dee6760 from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:43777)
I20240501 08:13:22.220816 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet b2e7c8c1795d4c6d997c75e63dee6760 from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:43777: session id = 4d516916eeac468db514eeb928f06190-b2e7c8c1795d4c6d997c75e63dee6760
I20240501 08:13:22.223753 16045 tablet_copy_source_session.cc:205] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:22.225153 25479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e7c8c1795d4c6d997c75e63dee6760. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:22.230542 25479 tablet_copy_client.cc:783] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:22.230903 25479 tablet_copy_client.cc:647] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:22.235456 25479 tablet_copy_client.cc:523] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:22.238948 25479 tablet_bootstrap.cc:492] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:13:22.254300 25479 tablet_bootstrap.cc:492] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:22.255061 25479 tablet_bootstrap.cc:492] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:13:22.255349 25479 ts_tablet_manager.cc:1367] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240501 08:13:22.256157 25479 raft_consensus.cc:348] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:22.256511 25479 raft_consensus.cc:729] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:13:22.256762 25479 consensus_queue.cc:260] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:22.257406 25479 ts_tablet_manager.cc:1398] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:22.261214 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-b2e7c8c1795d4c6d997c75e63dee6760 received from {username='slave'} at 127.6.56.70:43777
I20240501 08:13:22.261324 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 4d516916eeac468db514eeb928f06190-b2e7c8c1795d4c6d997c75e63dee6760 on tablet b2e7c8c1795d4c6d997c75e63dee6760 with peer 4d516916eeac468db514eeb928f06190
I20240501 08:13:22.285153 15755 consensus_queue.cc:237] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
W20240501 08:13:22.287778 15691 consensus_peers.cc:485] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e62b4a817174374a1ffdf9f8b25fb44. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:22.287812 16154 raft_consensus.cc:1261] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.287835 12974 raft_consensus.cc:1261] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.288480 24718 consensus_queue.cc:1035] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.288959 24718 consensus_queue.cc:1035] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.301430 24718 raft_consensus.cc:2932] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:22.302047 12974 raft_consensus.cc:2932] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:22.307180 16154 raft_consensus.cc:2932] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:22.310673 12833 catalog_manager.cc:5484] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:22.341040 25507 ts_tablet_manager.cc:902] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:13:22.356871 25507 tablet_copy_client.cc:308] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:13:22.357579 13254 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet 6fb95e3405ae45d3bfe71fb9d552e3fe from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:57331)
I20240501 08:13:22.357744 13254 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet 6fb95e3405ae45d3bfe71fb9d552e3fe from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:57331: session id = 982f49fc87f149439cda0de2261b6bfd-6fb95e3405ae45d3bfe71fb9d552e3fe
I20240501 08:13:22.360085 13254 tablet_copy_source_session.cc:205] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:22.364198 25507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fb95e3405ae45d3bfe71fb9d552e3fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:22.370036 25507 tablet_copy_client.cc:783] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:22.370334 25507 tablet_copy_client.cc:647] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:22.372407 25507 tablet_copy_client.cc:523] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:22.375453 25507 tablet_bootstrap.cc:492] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:13:22.415776 25523 ts_tablet_manager.cc:902] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:13:22.418237 25507 tablet_bootstrap.cc:492] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:22.418663 25507 tablet_bootstrap.cc:492] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:13:22.418985 25507 ts_tablet_manager.cc:1367] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.044s user 0.043s sys 0.001s
I20240501 08:13:22.419792 25507 raft_consensus.cc:348] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:22.420106 25507 raft_consensus.cc:729] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:13:22.420431 25507 consensus_queue.cc:260] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:22.421458 13234 consensus_queue.cc:237] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:22.425832 25507 ts_tablet_manager.cc:1398] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:13:22.430053 13254 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-6fb95e3405ae45d3bfe71fb9d552e3fe received from {username='slave'} at 127.6.56.69:57331
I20240501 08:13:22.430406 13254 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-6fb95e3405ae45d3bfe71fb9d552e3fe on tablet 6fb95e3405ae45d3bfe71fb9d552e3fe with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:13:22.440549 15892 raft_consensus.cc:1261] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.440893 16154 raft_consensus.cc:1261] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:22.441394 13170 consensus_peers.cc:485] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2b0c752d9614f2fbd3863d92df609a4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:22.441488 25355 consensus_queue.cc:1035] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.442198 25355 consensus_queue.cc:1035] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.454376 23695 raft_consensus.cc:2932] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:22.455037 16154 raft_consensus.cc:2932] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:22.457930 25532 ts_tablet_manager.cc:902] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:22.464494 25523 tablet_copy_client.cc:308] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.72:45383
I20240501 08:13:22.464499 12824 catalog_manager.cc:5484] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:22.465296 16178 tablet_copy_service.cc:140] P 6cc32f6e2d49477f9a23ce6e444039fb: Received BeginTabletCopySession request for tablet b000ec8b190644f2883e5f824463b762 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:52203)
I20240501 08:13:22.465426 16178 tablet_copy_service.cc:161] P 6cc32f6e2d49477f9a23ce6e444039fb: Beginning new tablet copy session on tablet b000ec8b190644f2883e5f824463b762 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:52203: session id = 4ecb724cb4764a53988fde00407db4c0-b000ec8b190644f2883e5f824463b762
I20240501 08:13:22.468096 16178 tablet_copy_source_session.cc:205] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:22.471994 15892 raft_consensus.cc:2932] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:22.479604 25532 tablet_copy_client.cc:308] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:22.480324 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 7a98388bf2fd402b8064d2cca06f5b8b from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:50899)
I20240501 08:13:22.480468 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 7a98388bf2fd402b8064d2cca06f5b8b from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:50899: session id = 6cc32f6e2d49477f9a23ce6e444039fb-7a98388bf2fd402b8064d2cca06f5b8b
I20240501 08:13:22.484436 25523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b000ec8b190644f2883e5f824463b762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:22.487216 13385 tablet_copy_source_session.cc:205] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:22.488554 25532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a98388bf2fd402b8064d2cca06f5b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:22.491498 25523 tablet_copy_client.cc:783] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:22.491709 25523 tablet_copy_client.cc:647] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:22.493197 25523 tablet_copy_client.cc:523] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:22.496260 25532 tablet_copy_client.cc:783] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:22.496497 25532 tablet_copy_client.cc:647] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:22.496917 25523 tablet_bootstrap.cc:492] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:22.498685 25532 tablet_copy_client.cc:523] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:22.502574 25532 tablet_bootstrap.cc:492] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:13:22.508531 25543 ts_tablet_manager.cc:902] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:13:22.518241 25523 tablet_bootstrap.cc:492] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:22.518810 25523 tablet_bootstrap.cc:492] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:22.519114 25523 ts_tablet_manager.cc:1367] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.003s
I20240501 08:13:22.520617 25523 raft_consensus.cc:348] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:22.520984 25523 raft_consensus.cc:729] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:22.521286 25523 consensus_queue.cc:260] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:22.531469 25523 ts_tablet_manager.cc:1398] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 08:13:22.533720 16178 tablet_copy_service.cc:342] P 6cc32f6e2d49477f9a23ce6e444039fb: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-b000ec8b190644f2883e5f824463b762 received from {username='slave'} at 127.6.56.68:52203
I20240501 08:13:22.533829 16178 tablet_copy_service.cc:433] P 6cc32f6e2d49477f9a23ce6e444039fb: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-b000ec8b190644f2883e5f824463b762 on tablet b000ec8b190644f2883e5f824463b762 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:22.572898 25532 tablet_bootstrap.cc:492] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:22.574667 25532 tablet_bootstrap.cc:492] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:13:22.574970 25532 ts_tablet_manager.cc:1367] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.072s user 0.062s sys 0.006s
I20240501 08:13:22.576071 25532 raft_consensus.cc:348] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:22.576503 25532 raft_consensus.cc:729] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:13:22.576745 25532 consensus_queue.cc:260] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:22.581972 25532 ts_tablet_manager.cc:1398] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:13:22.582685 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-7a98388bf2fd402b8064d2cca06f5b8b received from {username='slave'} at 127.6.56.72:50899
I20240501 08:13:22.582772 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-7a98388bf2fd402b8064d2cca06f5b8b on tablet 7a98388bf2fd402b8064d2cca06f5b8b with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:13:22.583314 25552 ts_tablet_manager.cc:902] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:13:22.598794 25543 tablet_copy_client.cc:308] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:13:22.599486 13124 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet 949d6765e68146d998ff4f5d2b120d68 from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:46045)
I20240501 08:13:22.599653 13124 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet 949d6765e68146d998ff4f5d2b120d68 from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:46045: session id = 76d830e199eb461c8ca689c9460ee0c2-949d6765e68146d998ff4f5d2b120d68
I20240501 08:13:22.602032 13124 tablet_copy_source_session.cc:205] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:22.605633 25543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 949d6765e68146d998ff4f5d2b120d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:22.642026 25551 ts_tablet_manager.cc:902] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:22.643303 25543 tablet_copy_client.cc:783] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:22.643608 25543 tablet_copy_client.cc:647] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:22.648157 25543 tablet_copy_client.cc:523] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:22.655416 25552 tablet_copy_client.cc:308] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.72:45383
I20240501 08:13:22.656148 16178 tablet_copy_service.cc:140] P 6cc32f6e2d49477f9a23ce6e444039fb: Received BeginTabletCopySession request for tablet 1eac68af4bc3473487bf91670cb48223 from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:38083)
I20240501 08:13:22.656332 16178 tablet_copy_service.cc:161] P 6cc32f6e2d49477f9a23ce6e444039fb: Beginning new tablet copy session on tablet 1eac68af4bc3473487bf91670cb48223 from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:38083: session id = 5dd722c770d34d6c9154a87917117161-1eac68af4bc3473487bf91670cb48223
I20240501 08:13:22.659786 16178 tablet_copy_source_session.cc:205] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:22.665326 25552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1eac68af4bc3473487bf91670cb48223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:22.668677 25543 tablet_bootstrap.cc:492] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:22.671314 25552 tablet_copy_client.cc:783] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:22.671516 25552 tablet_copy_client.cc:647] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:22.673031 25552 tablet_copy_client.cc:523] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:22.675783 25552 tablet_bootstrap.cc:492] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:13:22.683287 25551 tablet_copy_client.cc:308] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:22.684471 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 65a3d07808414c50a330da007c8f5e19 from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:58107)
I20240501 08:13:22.684646 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 65a3d07808414c50a330da007c8f5e19 from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:58107: session id = 6ed448d04e4646658c0de37d829ff15b-65a3d07808414c50a330da007c8f5e19
I20240501 08:13:22.688500 13385 tablet_copy_source_session.cc:205] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:22.690052 25551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65a3d07808414c50a330da007c8f5e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:22.691313 25552 tablet_bootstrap.cc:492] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:22.691778 25552 tablet_bootstrap.cc:492] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:13:22.692039 25552 ts_tablet_manager.cc:1367] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.010s
I20240501 08:13:22.692482 15892 consensus_queue.cc:237] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:22.693425 25552 raft_consensus.cc:348] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:22.693647 25552 raft_consensus.cc:729] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:13:22.693928 25552 consensus_queue.cc:260] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:22.694533 25552 ts_tablet_manager.cc:1398] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:13:22.696710 16178 tablet_copy_service.cc:342] P 6cc32f6e2d49477f9a23ce6e444039fb: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-1eac68af4bc3473487bf91670cb48223 received from {username='slave'} at 127.6.56.71:38083
I20240501 08:13:22.696818 16178 tablet_copy_service.cc:433] P 6cc32f6e2d49477f9a23ce6e444039fb: ending tablet copy session 5dd722c770d34d6c9154a87917117161-1eac68af4bc3473487bf91670cb48223 on tablet 1eac68af4bc3473487bf91670cb48223 with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:13:22.697247 25543 tablet_bootstrap.cc:492] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:22.697613 25543 tablet_bootstrap.cc:492] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:22.697902 25543 ts_tablet_manager.cc:1367] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.001s
I20240501 08:13:22.701814 25551 tablet_copy_client.cc:783] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:22.702056 25551 tablet_copy_client.cc:647] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:22.702684 25543 raft_consensus.cc:348] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:22.703297 25543 raft_consensus.cc:729] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:22.703578 25543 consensus_queue.cc:260] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:22.704201 25543 ts_tablet_manager.cc:1398] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:22.705050 15755 raft_consensus.cc:1261] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.705508 12972 raft_consensus.cc:1261] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:22.705627 15825 consensus_peers.cc:485] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9102cc8e9b92469ba2ccaa25fea2cc01. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:22.705828 24776 consensus_queue.cc:1035] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.706434 13124 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-949d6765e68146d998ff4f5d2b120d68 received from {username='slave'} at 127.6.56.65:46045
I20240501 08:13:22.706552 13124 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-949d6765e68146d998ff4f5d2b120d68 on tablet 949d6765e68146d998ff4f5d2b120d68 with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:13:22.708010 24776 consensus_queue.cc:1035] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.713732 25551 tablet_copy_client.cc:523] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:22.718959 25551 tablet_bootstrap.cc:492] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:13:22.738188 24776 raft_consensus.cc:2932] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:22.744753 25551 tablet_bootstrap.cc:492] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:22.744681 15755 raft_consensus.cc:2932] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:22.745364 25551 tablet_bootstrap.cc:492] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:13:22.745652 25551 ts_tablet_manager.cc:1367] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.008s
I20240501 08:13:22.746237 25551 raft_consensus.cc:348] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:22.746421 25551 raft_consensus.cc:729] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:13:22.746630 25551 consensus_queue.cc:260] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:22.747117 25551 ts_tablet_manager.cc:1398] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:13:22.749325 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-65a3d07808414c50a330da007c8f5e19 received from {username='slave'} at 127.6.56.67:58107
I20240501 08:13:22.749440 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-65a3d07808414c50a330da007c8f5e19 on tablet 65a3d07808414c50a330da007c8f5e19 with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:13:22.755004 15891 raft_consensus.cc:1203] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:22.758203 12833 catalog_manager.cc:5484] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:22.765607 12974 raft_consensus.cc:2932] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:22.806010 15891 consensus_queue.cc:237] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
W20240501 08:13:22.808504 15825 consensus_peers.cc:485] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39b12a1d835d428085d4a24233d78c38. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:22.808641 13235 raft_consensus.cc:1261] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.808874 15755 raft_consensus.cc:1261] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.809283 24776 consensus_queue.cc:1035] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.809666 25559 consensus_queue.cc:1035] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.812235 24776 raft_consensus.cc:2932] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:22.812930 15755 raft_consensus.cc:2932] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:22.815959 13235 raft_consensus.cc:2932] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:22.818917 12830 catalog_manager.cc:5484] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:22.840222 25523 ts_tablet_manager.cc:902] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:13:22.841856 25523 tablet_copy_client.cc:308] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:13:22.842473 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet 3e62b4a817174374a1ffdf9f8b25fb44 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:35425)
I20240501 08:13:22.842590 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet 3e62b4a817174374a1ffdf9f8b25fb44 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:35425: session id = 4ecb724cb4764a53988fde00407db4c0-3e62b4a817174374a1ffdf9f8b25fb44
I20240501 08:13:22.845086 15776 tablet_copy_source_session.cc:205] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:22.846318 15755 raft_consensus.cc:1203] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:13:22.846603 25523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e62b4a817174374a1ffdf9f8b25fb44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:22.853497 25523 tablet_copy_client.cc:783] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:22.853729 25523 tablet_copy_client.cc:647] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:22.855978 16154 raft_consensus.cc:1203] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:13:22.858820 25523 tablet_copy_client.cc:523] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:22.862066 25523 tablet_bootstrap.cc:492] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:22.865870 13365 consensus_queue.cc:237] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:22.868535 16024 raft_consensus.cc:1261] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.868777 13234 raft_consensus.cc:1261] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:22.869140 24312 consensus_queue.cc:1035] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:22.869799 13298 consensus_peers.cc:485] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6658ae0ccbcc4baf8853ab8778049030. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:22.870513 24312 consensus_queue.cc:1035] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:22.871567 25523 tablet_bootstrap.cc:492] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:22.871927 25523 tablet_bootstrap.cc:492] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:22.872160 25523 ts_tablet_manager.cc:1367] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240501 08:13:22.872861 25523 raft_consensus.cc:348] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:22.873070 25523 raft_consensus.cc:729] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:22.873312 25523 consensus_queue.cc:260] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:22.873948 25523 ts_tablet_manager.cc:1398] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 08:13:22.876331 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-3e62b4a817174374a1ffdf9f8b25fb44 received from {username='slave'} at 127.6.56.68:35425
I20240501 08:13:22.876479 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-3e62b4a817174374a1ffdf9f8b25fb44 on tablet 3e62b4a817174374a1ffdf9f8b25fb44 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:22.887706 25417 raft_consensus.cc:2932] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:22.888849 13234 raft_consensus.cc:2932] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:22.898357 16024 raft_consensus.cc:2932] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:22.900915 12833 catalog_manager.cc:5484] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:22.973171 13234 raft_consensus.cc:1203] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:22.978307 13365 raft_consensus.cc:1203] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:22.983178 12974 raft_consensus.cc:1203] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:23.000742 16024 consensus_queue.cc:237] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:23.003412 16154 raft_consensus.cc:1261] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:23.003543 15891 raft_consensus.cc:1261] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:23.003731 15959 consensus_peers.cc:485] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c8ec0d302644d378c54e4871f85159a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:23.004005 25377 consensus_queue.cc:1035] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:23.004271 25374 consensus_queue.cc:1035] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:23.009573 25377 raft_consensus.cc:2932] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:23.010391 15891 raft_consensus.cc:2932] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:23.015205 16154 raft_consensus.cc:2932] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:23.017467 12830 catalog_manager.cc:5484] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:23.032946 25650 ts_tablet_manager.cc:902] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:13:23.072909 25650 tablet_copy_client.cc:308] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:13:23.073676 13254 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet b2b0c752d9614f2fbd3863d92df609a4 from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:40993)
I20240501 08:13:23.073858 13254 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet b2b0c752d9614f2fbd3863d92df609a4 from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:40993: session id = e3fefd4dca2d41678187047f06c10187-b2b0c752d9614f2fbd3863d92df609a4
I20240501 08:13:23.076584 13254 tablet_copy_source_session.cc:205] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:23.076975 16154 consensus_queue.cc:237] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:23.079862 15755 raft_consensus.cc:1261] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:23.079967 12974 raft_consensus.cc:1261] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:23.080821 16092 consensus_peers.cc:485] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69d21bf5577f4b449a691ef6234376d2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:23.080891 25656 consensus_queue.cc:1035] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:23.081943 25396 consensus_queue.cc:1035] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:23.082806 25650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2b0c752d9614f2fbd3863d92df609a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:23.094442 25650 tablet_copy_client.cc:783] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:23.094686 25650 tablet_copy_client.cc:647] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:23.103924 25650 tablet_copy_client.cc:523] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:23.104434 25396 raft_consensus.cc:2932] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:23.107775 25650 tablet_bootstrap.cc:492] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:13:23.108772 15755 raft_consensus.cc:2932] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:23.115566 12833 catalog_manager.cc:5484] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:23.111873 12974 raft_consensus.cc:2932] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:23.149636 25650 tablet_bootstrap.cc:492] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:23.149984 25650 tablet_bootstrap.cc:492] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:13:23.150197 25650 ts_tablet_manager.cc:1367] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.013s
I20240501 08:13:23.150846 25650 raft_consensus.cc:348] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:23.151017 25650 raft_consensus.cc:729] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:13:23.151227 25650 consensus_queue.cc:260] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:23.153373 25650 ts_tablet_manager.cc:1398] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 08:13:23.153996 13254 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-b2b0c752d9614f2fbd3863d92df609a4 received from {username='slave'} at 127.6.56.66:40993
I20240501 08:13:23.154136 13254 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session e3fefd4dca2d41678187047f06c10187-b2b0c752d9614f2fbd3863d92df609a4 on tablet b2b0c752d9614f2fbd3863d92df609a4 with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:13:23.171540 25523 ts_tablet_manager.cc:902] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:13:23.184780 25523 tablet_copy_client.cc:308] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:13:23.185614 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 9102cc8e9b92469ba2ccaa25fea2cc01 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:37921)
I20240501 08:13:23.185786 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 9102cc8e9b92469ba2ccaa25fea2cc01 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:37921: session id = 4ecb724cb4764a53988fde00407db4c0-9102cc8e9b92469ba2ccaa25fea2cc01
I20240501 08:13:23.188483 15913 tablet_copy_source_session.cc:205] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:23.190090 25523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9102cc8e9b92469ba2ccaa25fea2cc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:23.193789 15755 consensus_queue.cc:237] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:23.196655 16154 raft_consensus.cc:1261] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:23.196751 12974 raft_consensus.cc:1261] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:23.197033 15691 consensus_peers.cc:485] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf908880298b462391a84f5a8d4e3a6b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:23.197265 25499 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:23.197559 25496 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:23.200130 25523 tablet_copy_client.cc:783] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:23.200393 25523 tablet_copy_client.cc:647] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:23.202347 25496 raft_consensus.cc:2932] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:23.202940 12974 raft_consensus.cc:2932] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:23.206651 16024 raft_consensus.cc:1203] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:23.210793 25523 tablet_copy_client.cc:523] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:23.215157 25523 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:23.218885 12824 catalog_manager.cc:5484] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:23.235519 16157 raft_consensus.cc:2932] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:23.246258 25523 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:23.246670 25523 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:23.246894 25523 ts_tablet_manager.cc:1367] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.002s
I20240501 08:13:23.249166 25523 raft_consensus.cc:348] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:23.249501 25523 raft_consensus.cc:729] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:23.249758 25523 consensus_queue.cc:260] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:23.251163 25523 ts_tablet_manager.cc:1398] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 08:13:23.257026 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-9102cc8e9b92469ba2ccaa25fea2cc01 received from {username='slave'} at 127.6.56.68:37921
I20240501 08:13:23.257172 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-9102cc8e9b92469ba2ccaa25fea2cc01 on tablet 9102cc8e9b92469ba2ccaa25fea2cc01 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:23.319909 25713 ts_tablet_manager.cc:902] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:13:23.385799 25718 ts_tablet_manager.cc:902] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:23.391813 25713 tablet_copy_client.cc:308] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:13:23.392418 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 39b12a1d835d428085d4a24233d78c38 from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:37437)
I20240501 08:13:23.392568 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 39b12a1d835d428085d4a24233d78c38 from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:37437: session id = 5dd722c770d34d6c9154a87917117161-39b12a1d835d428085d4a24233d78c38
I20240501 08:13:23.395336 15913 tablet_copy_source_session.cc:205] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:23.397032 25713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39b12a1d835d428085d4a24233d78c38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:23.404239 25718 tablet_copy_client.cc:308] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:23.405308 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 6658ae0ccbcc4baf8853ab8778049030 from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:50899)
I20240501 08:13:23.405472 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 6658ae0ccbcc4baf8853ab8778049030 from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:50899: session id = 6cc32f6e2d49477f9a23ce6e444039fb-6658ae0ccbcc4baf8853ab8778049030
I20240501 08:13:23.405570 25713 tablet_copy_client.cc:783] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:23.405781 25713 tablet_copy_client.cc:647] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:23.407888 13385 tablet_copy_source_session.cc:205] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:23.408957 25718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6658ae0ccbcc4baf8853ab8778049030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:23.409030 25713 tablet_copy_client.cc:523] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:23.412596 25713 tablet_bootstrap.cc:492] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:13:23.414983 25718 tablet_copy_client.cc:783] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:23.415205 25718 tablet_copy_client.cc:647] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:23.418184 25713 tablet_bootstrap.cc:492] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:23.418494 25713 tablet_bootstrap.cc:492] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:13:23.418718 25713 ts_tablet_manager.cc:1367] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 08:13:23.419575 25713 raft_consensus.cc:348] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:23.419792 25713 raft_consensus.cc:729] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:13:23.420436 25713 consensus_queue.cc:260] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:23.421001 25713 ts_tablet_manager.cc:1398] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:23.422260 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-39b12a1d835d428085d4a24233d78c38 received from {username='slave'} at 127.6.56.71:37437
I20240501 08:13:23.422386 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 5dd722c770d34d6c9154a87917117161-39b12a1d835d428085d4a24233d78c38 on tablet 39b12a1d835d428085d4a24233d78c38 with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:13:23.426112 25718 tablet_copy_client.cc:523] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:23.430228 25718 tablet_bootstrap.cc:492] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:13:23.445245 25718 tablet_bootstrap.cc:492] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:23.445672 25718 tablet_bootstrap.cc:492] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:13:23.445915 25718 ts_tablet_manager.cc:1367] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20240501 08:13:23.446347 13365 raft_consensus.cc:1203] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:23.446532 25718 raft_consensus.cc:348] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:23.446774 25718 raft_consensus.cc:729] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:13:23.446998 25718 consensus_queue.cc:260] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:23.447402 25718 ts_tablet_manager.cc:1398] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:13:23.450711 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-6658ae0ccbcc4baf8853ab8778049030 received from {username='slave'} at 127.6.56.72:50899
I20240501 08:13:23.450824 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-6658ae0ccbcc4baf8853ab8778049030 on tablet 6658ae0ccbcc4baf8853ab8778049030 with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:13:23.494052 13103 raft_consensus.cc:1203] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:23.567515 25523 ts_tablet_manager.cc:902] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:13:23.607383 25523 tablet_copy_client.cc:308] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.72:45383
I20240501 08:13:23.609016 16178 tablet_copy_service.cc:140] P 6cc32f6e2d49477f9a23ce6e444039fb: Received BeginTabletCopySession request for tablet 69d21bf5577f4b449a691ef6234376d2 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:52203)
I20240501 08:13:23.609151 16178 tablet_copy_service.cc:161] P 6cc32f6e2d49477f9a23ce6e444039fb: Beginning new tablet copy session on tablet 69d21bf5577f4b449a691ef6234376d2 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:52203: session id = 4ecb724cb4764a53988fde00407db4c0-69d21bf5577f4b449a691ef6234376d2
I20240501 08:13:23.612227 16178 tablet_copy_source_session.cc:205] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:23.618729 25523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d21bf5577f4b449a691ef6234376d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:23.622637 25650 ts_tablet_manager.cc:902] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:23.636353 25523 tablet_copy_client.cc:783] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:23.636770 25523 tablet_copy_client.cc:647] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:23.638346 25523 tablet_copy_client.cc:523] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:23.641148 25650 tablet_copy_client.cc:308] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:13:23.641803 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 3c8ec0d302644d378c54e4871f85159a from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:60749)
I20240501 08:13:23.641958 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 3c8ec0d302644d378c54e4871f85159a from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:60749: session id = e3fefd4dca2d41678187047f06c10187-3c8ec0d302644d378c54e4871f85159a
I20240501 08:13:23.642088 25523 tablet_bootstrap.cc:492] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:23.644657 16045 tablet_copy_source_session.cc:205] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:23.645830 25650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c8ec0d302644d378c54e4871f85159a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:23.654842 25650 tablet_copy_client.cc:783] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:23.655067 25650 tablet_copy_client.cc:647] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:23.656863 25650 tablet_copy_client.cc:523] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:23.657253 25523 tablet_bootstrap.cc:492] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:23.657636 25523 tablet_bootstrap.cc:492] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:23.657886 25523 ts_tablet_manager.cc:1367] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20240501 08:13:23.658622 25523 raft_consensus.cc:348] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:23.658857 25523 raft_consensus.cc:729] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:23.659083 25523 consensus_queue.cc:260] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:23.659704 25523 ts_tablet_manager.cc:1398] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:13:23.660703 25650 tablet_bootstrap.cc:492] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:13:23.661942 16178 tablet_copy_service.cc:342] P 6cc32f6e2d49477f9a23ce6e444039fb: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-69d21bf5577f4b449a691ef6234376d2 received from {username='slave'} at 127.6.56.68:52203
I20240501 08:13:23.662076 16178 tablet_copy_service.cc:433] P 6cc32f6e2d49477f9a23ce6e444039fb: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-69d21bf5577f4b449a691ef6234376d2 on tablet 69d21bf5577f4b449a691ef6234376d2 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:23.671077 25650 tablet_bootstrap.cc:492] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:23.671442 25650 tablet_bootstrap.cc:492] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:13:23.671667 25650 ts_tablet_manager.cc:1367] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20240501 08:13:23.672407 25650 raft_consensus.cc:348] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:23.672631 25650 raft_consensus.cc:729] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:13:23.672856 25650 consensus_queue.cc:260] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:23.673346 25650 ts_tablet_manager.cc:1398] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:23.674132 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-3c8ec0d302644d378c54e4871f85159a received from {username='slave'} at 127.6.56.66:60749
I20240501 08:13:23.674247 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session e3fefd4dca2d41678187047f06c10187-3c8ec0d302644d378c54e4871f85159a on tablet 3c8ec0d302644d378c54e4871f85159a with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:13:23.707257 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:23.761876 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:23.782189 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:23.789208 13363 raft_consensus.cc:1203] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:23.808045 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:23.809417 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:23.823915 25713 ts_tablet_manager.cc:902] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:13:23.825409 25713 tablet_copy_client.cc:308] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:13:23.827502 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet cf908880298b462391a84f5a8d4e3a6b from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:48491)
I20240501 08:13:23.827641 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet cf908880298b462391a84f5a8d4e3a6b from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:48491: session id = 5dd722c770d34d6c9154a87917117161-cf908880298b462391a84f5a8d4e3a6b
I20240501 08:13:23.830418 15776 tablet_copy_source_session.cc:205] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:23.832479 25713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf908880298b462391a84f5a8d4e3a6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:23.860174 16024 raft_consensus.cc:1203] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:23.877921 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:23.913517 16157 raft_consensus.cc:1203] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:23.941965 25713 tablet_copy_client.cc:783] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:23.944121 25713 tablet_copy_client.cc:647] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:23.946348 25713 tablet_copy_client.cc:523] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:23.968061 25713 tablet_bootstrap.cc:492] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:13:24.008328 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:24.016813 25713 tablet_bootstrap.cc:492] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:24.017200 25713 tablet_bootstrap.cc:492] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:13:24.017467 25713 ts_tablet_manager.cc:1367] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.000s
I20240501 08:13:24.018146 25713 raft_consensus.cc:348] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:24.018363 25713 raft_consensus.cc:729] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:13:24.018707 25713 consensus_queue.cc:260] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:24.020483 25713 ts_tablet_manager.cc:1398] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:13:24.028075 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:24.029503 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-cf908880298b462391a84f5a8d4e3a6b received from {username='slave'} at 127.6.56.71:48491
I20240501 08:13:24.029628 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session 5dd722c770d34d6c9154a87917117161-cf908880298b462391a84f5a8d4e3a6b on tablet cf908880298b462391a84f5a8d4e3a6b with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:13:24.114576 13363 raft_consensus.cc:1203] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:24.242199 13103 raft_consensus.cc:1203] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:24.386708 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:24.407217 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:24.417177 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:24.437170 16022 raft_consensus.cc:1203] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:24.459725 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:24.482347 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:24.509615 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:24.517714 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:24.538888 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:25.076145 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:25.153120 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:25.241801 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
W20240501 08:13:25.308851 15959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 153463us
I20240501 08:13:25.339552 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:25.350149 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 08:13:25.443487 12909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 193792us
I20240501 08:13:25.508539 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:25.575340 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:25.611871 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:25.726912 15891 consensus_queue.cc:237] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:25.729220 16022 raft_consensus.cc:1261] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:25.730707 25928 consensus_queue.cc:1035] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:25.731879 13235 raft_consensus.cc:1261] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:25.733197 25928 consensus_queue.cc:1035] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:25.733640 25850 raft_consensus.cc:2932] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:25.734234 16022 raft_consensus.cc:2932] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:25.741430 12825 catalog_manager.cc:5484] T 39b12a1d835d428085d4a24233d78c38 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:25.751224 13235 raft_consensus.cc:2932] T 39b12a1d835d428085d4a24233d78c38 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:25.766378 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 39b12a1d835d428085d4a24233d78c38 with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40166
I20240501 08:13:25.769260 25964 tablet_replica.cc:330] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:13:25.769759 25964 raft_consensus.cc:2229] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:25.769979 25964 raft_consensus.cc:2258] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:25.770534 25964 ts_tablet_manager.cc:1875] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:25.776732 25964 ts_tablet_manager.cc:1888] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:25.776854 25964 log.cc:1196] T 39b12a1d835d428085d4a24233d78c38 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/39b12a1d835d428085d4a24233d78c38
I20240501 08:13:25.778092 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 39b12a1d835d428085d4a24233d78c38 (table rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]) successfully deleted
I20240501 08:13:26.122069 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:26.169113 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:26.172310 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:26.188520 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:26.206923 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:26.231196 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:26.236251 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:26.269500 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
W20240501 08:13:26.278936 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 64149us
I20240501 08:13:26.455873 13235 consensus_queue.cc:237] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:26.458081 13103 raft_consensus.cc:1261] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:26.458274 16154 raft_consensus.cc:1261] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:26.459085 25903 consensus_queue.cc:1035] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:26.459734 25903 consensus_queue.cc:1035] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:26.466617 26046 raft_consensus.cc:2932] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:26.467159 16154 raft_consensus.cc:2932] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:26.468920 13103 raft_consensus.cc:2932] T b2b0c752d9614f2fbd3863d92df609a4 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:26.473083 12826 catalog_manager.cc:5484] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:26.481017 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet b2b0c752d9614f2fbd3863d92df609a4 with delete_type TABLET_DATA_TOMBSTONED (TS 4d516916eeac468db514eeb928f06190 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44232
I20240501 08:13:26.481931 26052 tablet_replica.cc:330] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:13:26.482231 26052 raft_consensus.cc:2229] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:26.483114 26052 raft_consensus.cc:2258] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:26.484000 26052 ts_tablet_manager.cc:1875] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:26.491109 26052 ts_tablet_manager.cc:1888] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:26.491248 26052 log.cc:1196] T b2b0c752d9614f2fbd3863d92df609a4 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/b2b0c752d9614f2fbd3863d92df609a4
I20240501 08:13:26.491942 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet b2b0c752d9614f2fbd3863d92df609a4 (table rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]) successfully deleted
I20240501 08:13:26.770931 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:26.821069 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:26.834806 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:26.839537 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:26.856865 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:26.878093 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:26.911275 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:26.951435 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:27.113956 15754 consensus_queue.cc:237] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:27.115837 13365 raft_consensus.cc:1261] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:27.116183 12973 raft_consensus.cc:1261] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:27.116746 25695 consensus_queue.cc:1035] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:27.117007 25499 consensus_queue.cc:1035] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:27.119843 25695 raft_consensus.cc:2932] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:27.120422 12973 raft_consensus.cc:2932] T 3e62b4a817174374a1ffdf9f8b25fb44 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:27.120771 13365 raft_consensus.cc:2932] T 3e62b4a817174374a1ffdf9f8b25fb44 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:27.125510 12828 catalog_manager.cc:5484] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:27.143491 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e62b4a817174374a1ffdf9f8b25fb44 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:13:27.144110 26144 tablet_replica.cc:330] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:13:27.144356 26144 raft_consensus.cc:2229] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:27.144589 26144 raft_consensus.cc:2258] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:27.145315 26144 ts_tablet_manager.cc:1875] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:27.151960 26144 ts_tablet_manager.cc:1888] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:27.152088 26144 log.cc:1196] T 3e62b4a817174374a1ffdf9f8b25fb44 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/3e62b4a817174374a1ffdf9f8b25fb44
I20240501 08:13:27.153117 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 3e62b4a817174374a1ffdf9f8b25fb44 (table rebalancer_extra_table_0 [id=63494fd5fa1e4f35ad51fa9669197a38]) successfully deleted
I20240501 08:13:27.453881 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:27.463414 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:27.467787 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:27.471027 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:27.477898 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:27.528553 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:27.553062 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:27.557623 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 08:13:27.580175 15825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 73242.9us
I20240501 08:13:27.727906 13365 consensus_queue.cc:237] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:27.730329 16157 raft_consensus.cc:1261] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:27.730868 26162 consensus_queue.cc:1035] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:27.731443 13234 raft_consensus.cc:1261] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:27.732127 26162 consensus_queue.cc:1035] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:27.733989 26162 raft_consensus.cc:2932] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:27.735668 16157 raft_consensus.cc:2932] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:27.736455 13234 raft_consensus.cc:2932] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:27.743088 12832 catalog_manager.cc:5484] T 65a3d07808414c50a330da007c8f5e19 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:27.754108 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet 65a3d07808414c50a330da007c8f5e19 with delete_type TABLET_DATA_TOMBSTONED (TS 4d516916eeac468db514eeb928f06190 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44232
I20240501 08:13:27.754710 26221 tablet_replica.cc:330] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:13:27.754963 26221 raft_consensus.cc:2229] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:27.755152 26221 raft_consensus.cc:2258] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:27.755667 26221 ts_tablet_manager.cc:1875] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:27.760627 26221 ts_tablet_manager.cc:1888] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:27.760773 26221 log.cc:1196] T 65a3d07808414c50a330da007c8f5e19 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/65a3d07808414c50a330da007c8f5e19
I20240501 08:13:27.761530 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 65a3d07808414c50a330da007c8f5e19 (table rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]) successfully deleted
I20240501 08:13:28.055981 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:28.077095 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:28.121197 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:28.128580 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:28.186249 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:28.186275 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:28.186663 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:28.260849 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:28.364449 16022 consensus_queue.cc:237] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:28.366350 13103 raft_consensus.cc:1261] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:28.366448 16157 raft_consensus.cc:1261] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:28.366959 25377 consensus_queue.cc:1035] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:28.367386 26185 consensus_queue.cc:1035] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:28.370164 25377 raft_consensus.cc:2932] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:28.370988 13103 raft_consensus.cc:2932] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:28.372329 16157 raft_consensus.cc:2932] T 3c8ec0d302644d378c54e4871f85159a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:28.377512 12827 catalog_manager.cc:5484] T 3c8ec0d302644d378c54e4871f85159a P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:28.384047 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c8ec0d302644d378c54e4871f85159a with delete_type TABLET_DATA_TOMBSTONED (TS 4d516916eeac468db514eeb928f06190 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44232
I20240501 08:13:28.386637 26302 tablet_replica.cc:330] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:13:28.386899 26302 raft_consensus.cc:2229] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:28.387130 26302 raft_consensus.cc:2258] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:28.387918 26302 ts_tablet_manager.cc:1875] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:28.396095 26302 ts_tablet_manager.cc:1888] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:28.396405 26302 log.cc:1196] T 3c8ec0d302644d378c54e4871f85159a P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/3c8ec0d302644d378c54e4871f85159a
I20240501 08:13:28.397476 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 3c8ec0d302644d378c54e4871f85159a (table rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]) successfully deleted
I20240501 08:13:28.745193 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:28.752532 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:28.763348 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:28.802667 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:28.813294 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:28.820142 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:28.827531 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:28.844609 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:29.054036 16157 consensus_queue.cc:237] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:29.055938 13365 raft_consensus.cc:1261] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:29.055953 16022 raft_consensus.cc:1261] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:29.056532 26349 consensus_queue.cc:1035] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:29.060701 26349 consensus_queue.cc:1035] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:29.062064 26194 raft_consensus.cc:2932] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:29.065305 13365 raft_consensus.cc:2932] T 1eac68af4bc3473487bf91670cb48223 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:29.068073 12828 catalog_manager.cc:5484] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:29.081161 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 1eac68af4bc3473487bf91670cb48223 with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40166
I20240501 08:13:29.082201 16022 raft_consensus.cc:2932] T 1eac68af4bc3473487bf91670cb48223 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:29.097657 26394 tablet_replica.cc:330] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:13:29.099176 26394 raft_consensus.cc:2229] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:29.099540 26394 raft_consensus.cc:2258] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:29.100359 26394 ts_tablet_manager.cc:1875] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:29.107095 26394 ts_tablet_manager.cc:1888] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:29.107218 26394 log.cc:1196] T 1eac68af4bc3473487bf91670cb48223 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/1eac68af4bc3473487bf91670cb48223
I20240501 08:13:29.108027 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 1eac68af4bc3473487bf91670cb48223 (table rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]) successfully deleted
I20240501 08:13:29.397984 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:29.444193 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:29.455355 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:29.455421 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:29.474473 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:29.511368 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:29.525167 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:29.557520 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:29.690599 15891 consensus_queue.cc:237] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:29.692281 13365 raft_consensus.cc:1261] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:29.692385 15754 raft_consensus.cc:1261] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:29.692812 26192 consensus_queue.cc:1035] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:29.693078 26452 consensus_queue.cc:1035] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:29.695691 26192 raft_consensus.cc:2932] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:29.698556 13363 raft_consensus.cc:2932] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:29.703934 15754 raft_consensus.cc:2932] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:29.706547 12826 catalog_manager.cc:5484] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:29.713060 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 9102cc8e9b92469ba2ccaa25fea2cc01 with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57838
I20240501 08:13:29.715131 26482 tablet_replica.cc:330] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:13:29.715409 26482 raft_consensus.cc:2229] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:29.716367 26482 raft_consensus.cc:2258] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:29.717272 26482 ts_tablet_manager.cc:1875] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:29.722609 26482 ts_tablet_manager.cc:1888] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:29.722793 26482 log.cc:1196] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/9102cc8e9b92469ba2ccaa25fea2cc01
I20240501 08:13:29.723599 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 9102cc8e9b92469ba2ccaa25fea2cc01 (table rebalancer_extra_table_30 [id=aa62935a584c4c33ad4e2e98639049fc]) successfully deleted
I20240501 08:13:29.941881 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:29.963239 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:30.032135 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:30.077534 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:30.078769 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:30.095826 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:30.099094 13084 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:30.123260 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:30.277356 13103 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68"
dest_uuid: "e3fefd4dca2d41678187047f06c10187"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46598
I20240501 08:13:30.277649 13103 raft_consensus.cc:595] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:30.540971 25955 raft_consensus.cc:979] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: : Instructing follower 982f49fc87f149439cda0de2261b6bfd to start an election
I20240501 08:13:30.541146 25955 raft_consensus.cc:1067] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Signalling peer 982f49fc87f149439cda0de2261b6bfd to start an election
I20240501 08:13:30.543221 15754 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68"
dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
from {username='slave'} at 127.6.56.66:33955
I20240501 08:13:30.543494 15754 raft_consensus.cc:482] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:30.543617 15754 raft_consensus.cc:3037] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:30.548946 15754 raft_consensus.cc:504] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:30.552835 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:13:30.553086 15892 raft_consensus.cc:3037] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:30.555913 15892 raft_consensus.cc:2454] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 2.
I20240501 08:13:30.569473 13103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:13:30.569754 13103 raft_consensus.cc:3032] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:30.569823 13103 raft_consensus.cc:729] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Running, Role: LEADER
I20240501 08:13:30.570130 13103 consensus_queue.cc:260] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:30.570540 13103 raft_consensus.cc:3037] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:30.580848 13103 raft_consensus.cc:2454] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 2.
I20240501 08:13:30.581502 15691 leader_election.cc:302] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd, e3fefd4dca2d41678187047f06c10187; no voters:
I20240501 08:13:30.581918 25695 raft_consensus.cc:2781] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:30.582046 25695 raft_consensus.cc:686] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 2 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:13:30.582298 25695 consensus_queue.cc:237] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:30.586995 12971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "949d6765e68146d998ff4f5d2b120d68" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
I20240501 08:13:30.587240 12971 raft_consensus.cc:3037] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:30.592962 12971 raft_consensus.cc:2454] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 2.
I20240501 08:13:30.594887 15754 leader_election.cc:288] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 election: Requested vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:13:30.603721 12831 catalog_manager.cc:5484] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 1 to 2, leader changed from e3fefd4dca2d41678187047f06c10187 (127.6.56.66) to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 2 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
W20240501 08:13:30.616268 12909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 57633.7us
I20240501 08:13:30.812083 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:30.819772 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:30.847731 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:30.872577 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:30.878206 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:30.879976 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:30.963305 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:30.992455 15892 raft_consensus.cc:1261] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:30.993480 25695 consensus_queue.cc:1035] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:31.001899 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:31.071493 13103 raft_consensus.cc:1261] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:31.072116 25695 consensus_queue.cc:1035] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:31.083423 12971 raft_consensus.cc:1261] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:31.083990 25695 consensus_queue.cc:1035] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:31.442409 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:31.452425 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:31.463268 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:31.474203 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:31.558287 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:31.585474 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:31.591338 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:31.623065 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:31.758584 13363 consensus_queue.cc:237] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:31.760514 12973 raft_consensus.cc:1261] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:31.760517 16157 raft_consensus.cc:1261] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:31.761098 25993 consensus_queue.cc:1035] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:31.761328 26162 consensus_queue.cc:1035] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:31.764385 26162 raft_consensus.cc:2932] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:31.765337 12973 raft_consensus.cc:2932] T 7a98388bf2fd402b8064d2cca06f5b8b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:31.765902 16157 raft_consensus.cc:2932] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:31.771399 12831 catalog_manager.cc:5484] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 4 to 5, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:31.786866 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a98388bf2fd402b8064d2cca06f5b8b with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:34524
I20240501 08:13:31.788790 26713 tablet_replica.cc:330] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:13:31.789072 26713 raft_consensus.cc:2229] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:31.789403 26713 raft_consensus.cc:2258] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:31.790239 26713 ts_tablet_manager.cc:1875] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:31.796981 26713 ts_tablet_manager.cc:1888] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:13:31.797130 26713 log.cc:1196] T 7a98388bf2fd402b8064d2cca06f5b8b P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/7a98388bf2fd402b8064d2cca06f5b8b
I20240501 08:13:31.797938 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 7a98388bf2fd402b8064d2cca06f5b8b (table rebalance_test_table_3 [id=9d3f2a7a052c418bbe2d3bb437760b4f]) successfully deleted
I20240501 08:13:32.243181 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:32.263022 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:32.263311 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:32.286948 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:32.290421 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:32.382632 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:32.430315 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:32.446705 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:32.604048 16157 consensus_queue.cc:237] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:32.606645 13363 raft_consensus.cc:1261] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:32.606865 13103 raft_consensus.cc:1261] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:32.607211 26785 consensus_queue.cc:1035] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:32.607653 26592 consensus_queue.cc:1035] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:32.610554 26592 raft_consensus.cc:2932] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:32.613337 13363 raft_consensus.cc:2932] T b000ec8b190644f2883e5f824463b762 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:32.613325 13103 raft_consensus.cc:2932] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:32.620030 12829 catalog_manager.cc:5484] T b000ec8b190644f2883e5f824463b762 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:32.627013 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet b000ec8b190644f2883e5f824463b762 with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:13:32.628908 26813 tablet_replica.cc:330] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:13:32.629585 26813 raft_consensus.cc:2229] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:32.629864 26813 raft_consensus.cc:2258] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:32.630605 26813 ts_tablet_manager.cc:1875] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:32.637945 26813 ts_tablet_manager.cc:1888] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:32.638170 26813 log.cc:1196] T b000ec8b190644f2883e5f824463b762 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/b000ec8b190644f2883e5f824463b762
I20240501 08:13:32.639156 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet b000ec8b190644f2883e5f824463b762 (table rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]) successfully deleted
I20240501 08:13:32.901211 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:32.937862 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:32.943883 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:32.956424 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:32.976450 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:32.985260 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:33.085851 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:33.096405 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:33.210327 16022 consensus_queue.cc:237] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:13:33.212891 15754 raft_consensus.cc:1261] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:33.213794 25377 consensus_queue.cc:1035] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:33.214851 15892 raft_consensus.cc:1261] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:33.215468 26784 consensus_queue.cc:1035] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:33.216657 25377 raft_consensus.cc:2932] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:33.217267 15754 raft_consensus.cc:2932] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:33.224555 12833 catalog_manager.cc:5484] T b2e7c8c1795d4c6d997c75e63dee6760 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:33.232045 15893 raft_consensus.cc:2932] T b2e7c8c1795d4c6d997c75e63dee6760 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:13:33.246026 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e7c8c1795d4c6d997c75e63dee6760 with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57838
I20240501 08:13:33.260622 26895 tablet_replica.cc:330] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:13:33.260849 26895 raft_consensus.cc:2229] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:33.261061 26895 raft_consensus.cc:2258] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:33.261636 26895 ts_tablet_manager.cc:1875] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:33.267575 26895 ts_tablet_manager.cc:1888] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:33.267680 26895 log.cc:1196] T b2e7c8c1795d4c6d997c75e63dee6760 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/b2e7c8c1795d4c6d997c75e63dee6760
I20240501 08:13:33.270568 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet b2e7c8c1795d4c6d997c75e63dee6760 (table rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]) successfully deleted
I20240501 08:13:33.509761 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:33.517113 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:33.525780 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:33.544580 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:33.577683 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:33.588745 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:13:33.592079 12907 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 89600.3us
I20240501 08:13:33.616701 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:33.681329 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:33.812775 13232 consensus_queue.cc:237] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:33.814877 15754 raft_consensus.cc:1261] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:33.814893 13103 raft_consensus.cc:1261] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:33.815452 25903 consensus_queue.cc:1035] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:33.815729 26255 consensus_queue.cc:1035] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:33.819227 25903 raft_consensus.cc:2932] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:33.820911 13103 raft_consensus.cc:2932] T 6fb95e3405ae45d3bfe71fb9d552e3fe P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:33.823278 15754 raft_consensus.cc:2932] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:33.825709 12833 catalog_manager.cc:5484] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:33.849840 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fb95e3405ae45d3bfe71fb9d552e3fe with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:13:33.850435 26983 tablet_replica.cc:330] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:13:33.850682 26983 raft_consensus.cc:2229] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:33.850930 26983 raft_consensus.cc:2258] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:33.851671 26983 ts_tablet_manager.cc:1875] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:33.857239 26983 ts_tablet_manager.cc:1888] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:13:33.857365 26983 log.cc:1196] T 6fb95e3405ae45d3bfe71fb9d552e3fe P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/6fb95e3405ae45d3bfe71fb9d552e3fe
I20240501 08:13:33.858021 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 6fb95e3405ae45d3bfe71fb9d552e3fe (table rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]) successfully deleted
I20240501 08:13:34.137431 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:34.155606 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:34.160068 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:34.168622 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:34.178789 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:34.217051 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:34.290445 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:34.347157 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:34.441808 15891 consensus_queue.cc:237] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:34.443783 13363 raft_consensus.cc:1261] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:34.443791 15754 raft_consensus.cc:1261] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:34.444279 26452 consensus_queue.cc:1035] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:34.444599 27035 consensus_queue.cc:1035] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:34.447499 27035 raft_consensus.cc:2932] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:34.448089 13363 raft_consensus.cc:2932] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:34.448089 15754 raft_consensus.cc:2932] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:34.455842 12826 catalog_manager.cc:5484] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:34.461887 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet f305a6e24ab1436b895db037f5d72870 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:13:34.472103 27061 tablet_replica.cc:330] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:13:34.472398 27061 raft_consensus.cc:2229] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:34.472683 27061 raft_consensus.cc:2258] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:34.473433 27061 ts_tablet_manager.cc:1875] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:34.479069 27061 ts_tablet_manager.cc:1888] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:34.479192 27061 log.cc:1196] T f305a6e24ab1436b895db037f5d72870 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/f305a6e24ab1436b895db037f5d72870
I20240501 08:13:34.479951 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet f305a6e24ab1436b895db037f5d72870 (table rebalancer_extra_table_38 [id=2da24d5d679d41708cbfac4594959235]) successfully deleted
I20240501 08:13:34.792830 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:34.819200 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:34.820712 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:34.855474 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:34.856776 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:34.869550 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:34.882297 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:34.888821 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:13:34.961885 13298 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 88274.1us
I20240501 08:13:35.067202 12822 maintenance_manager.cc:419] P c48be6e775954c7093aaf62e68805ec0: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033380
I20240501 08:13:35.134982 12821 maintenance_manager.cc:643] P c48be6e775954c7093aaf62e68805ec0: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.067s user 0.044s sys 0.019s Metrics: {"bytes_written":89759,"cfile_init":1,"dirs.queue_time_us":6373,"dirs.run_cpu_time_us":238,"dirs.run_wall_time_us":1892,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1519,"lbm_writes_lt_1ms":33,"peak_mem_usage":0,"rows_written":249,"thread_start_us":6580,"threads_started":1}
I20240501 08:13:35.136183 12822 maintenance_manager.cc:419] P c48be6e775954c7093aaf62e68805ec0: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 29315 bytes on disk
I20240501 08:13:35.137291 12821 maintenance_manager.cc:643] P c48be6e775954c7093aaf62e68805ec0: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 08:13:35.147639 13365 consensus_queue.cc:237] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:35.149602 16022 raft_consensus.cc:1261] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:35.149626 16154 raft_consensus.cc:1261] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:35.150189 27126 consensus_queue.cc:1035] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:35.150455 27014 consensus_queue.cc:1035] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:35.153599 27014 raft_consensus.cc:2932] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:35.154196 16154 raft_consensus.cc:2932] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:35.154439 16022 raft_consensus.cc:2932] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:35.160427 12828 catalog_manager.cc:5484] T 6658ae0ccbcc4baf8853ab8778049030 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:35.176106 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet 6658ae0ccbcc4baf8853ab8778049030 with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34524
I20240501 08:13:35.176777 27151 tablet_replica.cc:330] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:13:35.176975 27151 raft_consensus.cc:2229] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:35.177151 27151 raft_consensus.cc:2258] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:35.177713 27151 ts_tablet_manager.cc:1875] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:35.183343 27151 ts_tablet_manager.cc:1888] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:35.183547 27151 log.cc:1196] T 6658ae0ccbcc4baf8853ab8778049030 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/6658ae0ccbcc4baf8853ab8778049030
I20240501 08:13:35.184644 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 6658ae0ccbcc4baf8853ab8778049030 (table rebalancer_extra_table_54 [id=6bbde8949c684c9e8a921fd00ec78595]) successfully deleted
I20240501 08:13:35.497035 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:35.503757 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:35.528096 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:35.559834 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:35.595472 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:35.599053 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:35.603682 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:35.606076 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:35.780860 16157 consensus_queue.cc:237] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:35.782950 13365 raft_consensus.cc:1261] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:35.783152 15754 raft_consensus.cc:1261] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:35.783720 27241 consensus_queue.cc:1035] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:35.784044 27127 consensus_queue.cc:1035] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:35.787288 27127 raft_consensus.cc:2932] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:35.787609 13365 raft_consensus.cc:2932] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:35.789588 15754 raft_consensus.cc:2932] T 69d21bf5577f4b449a691ef6234376d2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:35.794651 12830 catalog_manager.cc:5484] T 69d21bf5577f4b449a691ef6234376d2 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:35.819960 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d21bf5577f4b449a691ef6234376d2 with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57838
I20240501 08:13:35.824695 27254 tablet_replica.cc:330] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:13:35.824961 27254 raft_consensus.cc:2229] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:35.825165 27254 raft_consensus.cc:2258] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:35.825776 27254 ts_tablet_manager.cc:1875] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:35.831491 27254 ts_tablet_manager.cc:1888] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:35.831621 27254 log.cc:1196] T 69d21bf5577f4b449a691ef6234376d2 P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/69d21bf5577f4b449a691ef6234376d2
I20240501 08:13:35.832547 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 69d21bf5577f4b449a691ef6234376d2 (table rebalancer_extra_table_6 [id=44cbc312bbfb43ea9e04c46ba14ba097]) successfully deleted
I20240501 08:13:35.883018 16022 consensus_queue.cc:237] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:35.885572 13232 raft_consensus.cc:1261] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:35.886276 25377 consensus_queue.cc:1035] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:35.887356 15959 consensus_peers.cc:485] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca6339e48a2f4b9e94e32e05c7bf343b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:35.888270 12973 raft_consensus.cc:1261] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:35.889035 27221 consensus_queue.cc:1035] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:35.891583 25377 raft_consensus.cc:2932] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:35.892199 12973 raft_consensus.cc:2932] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:35.897961 13232 raft_consensus.cc:2932] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:35.898320 12824 catalog_manager.cc:5484] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:36.015107 16022 consensus_queue.cc:237] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:36.017176 15892 raft_consensus.cc:1261] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:36.017331 13232 raft_consensus.cc:1261] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:36.017824 27221 consensus_queue.cc:1035] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:36.018196 27099 consensus_queue.cc:1035] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:36.018568 15959 consensus_peers.cc:485] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 279b5f4d8fba4c12beec040c6ed39992. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:36.020906 27099 raft_consensus.cc:2932] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:36.021471 13232 raft_consensus.cc:2932] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:36.027364 12830 catalog_manager.cc:5484] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:36.030205 15892 raft_consensus.cc:2932] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:36.082441 13105 consensus_queue.cc:237] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:36.084988 16022 raft_consensus.cc:1261] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:36.085107 16154 raft_consensus.cc:1261] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:36.085566 27318 consensus_queue.cc:1035] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:36.085820 26751 consensus_queue.cc:1035] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:36.086057 13041 consensus_peers.cc:485] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21719a43fa6845f8a3a5337dbe770f60. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:36.090385 26751 raft_consensus.cc:2932] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:36.091138 16022 raft_consensus.cc:2932] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:36.099172 12824 catalog_manager.cc:5484] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:36.102319 16154 raft_consensus.cc:2932] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:36.144762 13365 consensus_queue.cc:237] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:36.147102 12973 raft_consensus.cc:1261] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:36.147130 13105 raft_consensus.cc:1261] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:13:36.147209 13298 consensus_peers.cc:485] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a328773ef57444cb8147524107b15dba. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:36.147688 27014 consensus_queue.cc:1035] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:36.147914 26162 consensus_queue.cc:1035] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:36.152570 27014 raft_consensus.cc:2932] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:36.153126 12973 raft_consensus.cc:2932] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:36.159332 12824 catalog_manager.cc:5484] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:36.163111 13105 raft_consensus.cc:2932] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:36.451510 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:36.460081 27398 ts_tablet_manager.cc:902] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:36.475134 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:36.475368 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:36.505396 27398 tablet_copy_client.cc:308] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:13:36.506238 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet ca6339e48a2f4b9e94e32e05c7bf343b from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:60749)
I20240501 08:13:36.506420 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet ca6339e48a2f4b9e94e32e05c7bf343b from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:60749: session id = e3fefd4dca2d41678187047f06c10187-ca6339e48a2f4b9e94e32e05c7bf343b
I20240501 08:13:36.506580 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:36.508989 16045 tablet_copy_source_session.cc:205] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:36.510671 27398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca6339e48a2f4b9e94e32e05c7bf343b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.525626 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:36.526813 27398 tablet_copy_client.cc:783] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:36.527137 27398 tablet_copy_client.cc:647] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:36.530138 27398 tablet_copy_client.cc:523] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:36.534379 27398 tablet_bootstrap.cc:492] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:13:36.588513 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:36.604329 27398 tablet_bootstrap.cc:492] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:36.605861 27398 tablet_bootstrap.cc:492] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:13:36.606616 27398 ts_tablet_manager.cc:1367] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.011s
I20240501 08:13:36.609815 27398 raft_consensus.cc:348] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:36.610589 27398 raft_consensus.cc:729] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.611197 27410 ts_tablet_manager.cc:902] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:13:36.611588 27398 consensus_queue.cc:260] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:36.614323 27398 ts_tablet_manager.cc:1398] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240501 08:13:36.622488 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-ca6339e48a2f4b9e94e32e05c7bf343b received from {username='slave'} at 127.6.56.66:60749
I20240501 08:13:36.622637 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session e3fefd4dca2d41678187047f06c10187-ca6339e48a2f4b9e94e32e05c7bf343b on tablet ca6339e48a2f4b9e94e32e05c7bf343b with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:13:36.630882 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:36.640900 27398 ts_tablet_manager.cc:902] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:36.663898 27410 tablet_copy_client.cc:308] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:13:36.664634 13124 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet 21719a43fa6845f8a3a5337dbe770f60 from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:33769)
I20240501 08:13:36.664806 13124 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet 21719a43fa6845f8a3a5337dbe770f60 from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:33769: session id = 6ed448d04e4646658c0de37d829ff15b-21719a43fa6845f8a3a5337dbe770f60
I20240501 08:13:36.667798 13124 tablet_copy_source_session.cc:205] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:36.671416 27410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21719a43fa6845f8a3a5337dbe770f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.676268 27398 tablet_copy_client.cc:308] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:13:36.677228 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 279b5f4d8fba4c12beec040c6ed39992 from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:60749)
I20240501 08:13:36.677392 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 279b5f4d8fba4c12beec040c6ed39992 from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:60749: session id = e3fefd4dca2d41678187047f06c10187-279b5f4d8fba4c12beec040c6ed39992
I20240501 08:13:36.680112 16045 tablet_copy_source_session.cc:205] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:36.683068 27398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279b5f4d8fba4c12beec040c6ed39992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.717692 27398 tablet_copy_client.cc:783] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:36.728556 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:36.719074 27398 tablet_copy_client.cc:647] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:36.738544 27398 tablet_copy_client.cc:523] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:36.756270 27398 tablet_bootstrap.cc:492] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:13:36.762949 27410 tablet_copy_client.cc:783] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:36.763227 27410 tablet_copy_client.cc:647] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:36.765236 27410 tablet_copy_client.cc:523] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:36.777870 27410 tablet_bootstrap.cc:492] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:13:36.782745 27420 ts_tablet_manager.cc:902] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:36.788813 27410 tablet_bootstrap.cc:492] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:36.789291 27410 tablet_bootstrap.cc:492] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:13:36.789584 27410 ts_tablet_manager.cc:1367] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 08:13:36.790318 27410 raft_consensus.cc:348] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:36.790643 27410 raft_consensus.cc:729] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.790920 27410 consensus_queue.cc:260] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:36.793800 27410 ts_tablet_manager.cc:1398] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:13:36.794450 13124 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-21719a43fa6845f8a3a5337dbe770f60 received from {username='slave'} at 127.6.56.67:33769
I20240501 08:13:36.794564 13124 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-21719a43fa6845f8a3a5337dbe770f60 on tablet 21719a43fa6845f8a3a5337dbe770f60 with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:13:36.795995 27398 tablet_bootstrap.cc:492] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:36.796689 27398 tablet_bootstrap.cc:492] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:13:36.798122 27398 ts_tablet_manager.cc:1367] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.042s user 0.006s sys 0.004s
I20240501 08:13:36.800864 27398 raft_consensus.cc:348] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:36.801194 27398 raft_consensus.cc:729] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.801496 27398 consensus_queue.cc:260] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:36.802453 27398 ts_tablet_manager.cc:1398] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:13:36.803723 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-279b5f4d8fba4c12beec040c6ed39992 received from {username='slave'} at 127.6.56.66:60749
I20240501 08:13:36.803846 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session e3fefd4dca2d41678187047f06c10187-279b5f4d8fba4c12beec040c6ed39992 on tablet 279b5f4d8fba4c12beec040c6ed39992 with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:13:36.830647 27420 tablet_copy_client.cc:308] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:36.831333 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet a328773ef57444cb8147524107b15dba from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:50899)
I20240501 08:13:36.831519 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet a328773ef57444cb8147524107b15dba from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:50899: session id = 6cc32f6e2d49477f9a23ce6e444039fb-a328773ef57444cb8147524107b15dba
I20240501 08:13:36.833575 13385 tablet_copy_source_session.cc:205] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:36.834758 27420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a328773ef57444cb8147524107b15dba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:36.839671 27420 tablet_copy_client.cc:783] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:36.839892 27420 tablet_copy_client.cc:647] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:36.841393 27420 tablet_copy_client.cc:523] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:36.844801 27420 tablet_bootstrap.cc:492] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:13:36.865865 13103 raft_consensus.cc:1203] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:36.890573 27420 tablet_bootstrap.cc:492] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:36.891018 27420 tablet_bootstrap.cc:492] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:13:36.891291 27420 ts_tablet_manager.cc:1367] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.047s user 0.040s sys 0.000s
I20240501 08:13:36.891968 27420 raft_consensus.cc:348] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:36.892211 27420 raft_consensus.cc:729] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:13:36.892464 27420 consensus_queue.cc:260] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:36.893133 27420 ts_tablet_manager.cc:1398] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 08:13:36.894364 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-a328773ef57444cb8147524107b15dba received from {username='slave'} at 127.6.56.72:50899
I20240501 08:13:36.894490 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-a328773ef57444cb8147524107b15dba on tablet a328773ef57444cb8147524107b15dba with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:13:37.078830 13234 raft_consensus.cc:1203] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:37.222241 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:37.229353 13103 raft_consensus.cc:1203] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:37.266549 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:37.274682 16154 raft_consensus.cc:1203] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:13:37.292634 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:37.319830 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:37.364177 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:37.377192 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:37.455051 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:37.479156 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:37.622197 13103 consensus_queue.cc:237] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:37.624014 16154 raft_consensus.cc:1261] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:37.624087 13232 raft_consensus.cc:1261] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:37.624554 27484 consensus_queue.cc:1035] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:37.624859 27479 consensus_queue.cc:1035] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:37.627313 27479 raft_consensus.cc:2932] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:37.627887 16154 raft_consensus.cc:2932] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:37.630780 13232 raft_consensus.cc:2932] T 21719a43fa6845f8a3a5337dbe770f60 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:37.633265 12831 catalog_manager.cc:5484] T 21719a43fa6845f8a3a5337dbe770f60 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:37.647214 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 21719a43fa6845f8a3a5337dbe770f60 with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:13:37.648236 27532 tablet_replica.cc:330] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:13:37.648520 27532 raft_consensus.cc:2229] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:37.648736 27532 raft_consensus.cc:2258] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:37.649291 27532 ts_tablet_manager.cc:1875] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:37.658103 27532 ts_tablet_manager.cc:1888] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:37.658250 27532 log.cc:1196] T 21719a43fa6845f8a3a5337dbe770f60 P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/21719a43fa6845f8a3a5337dbe770f60
I20240501 08:13:37.658947 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 21719a43fa6845f8a3a5337dbe770f60 (table rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]) successfully deleted
W20240501 08:13:37.856998 12909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 50741.3us
I20240501 08:13:37.879163 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:37.886282 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:37.931043 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:37.974499 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:37.994114 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:38.056069 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:38.137164 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:38.242709 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:38.443302 16024 consensus_queue.cc:237] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:38.445896 13232 raft_consensus.cc:1261] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:38.445916 13103 raft_consensus.cc:1261] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:38.446550 27587 consensus_queue.cc:1035] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:38.446837 27099 consensus_queue.cc:1035] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:38.449430 27099 raft_consensus.cc:2932] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:38.450050 13232 raft_consensus.cc:2932] T 279b5f4d8fba4c12beec040c6ed39992 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:38.453027 13103 raft_consensus.cc:2932] T 279b5f4d8fba4c12beec040c6ed39992 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:38.459445 12827 catalog_manager.cc:5484] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 2 to 3, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:38.497503 15872 tablet_service.cc:1514] Processing DeleteTablet for tablet 279b5f4d8fba4c12beec040c6ed39992 with delete_type TABLET_DATA_TOMBSTONED (TS 4d516916eeac468db514eeb928f06190 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:44232
I20240501 08:13:38.500226 27616 tablet_replica.cc:330] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190: stopping tablet replica
I20240501 08:13:38.500443 27616 raft_consensus.cc:2229] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:38.500639 27616 raft_consensus.cc:2258] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:38.501237 27616 ts_tablet_manager.cc:1875] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:38.506806 27616 ts_tablet_manager.cc:1888] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:38.507009 27616 log.cc:1196] T 279b5f4d8fba4c12beec040c6ed39992 P 4d516916eeac468db514eeb928f06190: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/wals/279b5f4d8fba4c12beec040c6ed39992
I20240501 08:13:38.508138 12817 catalog_manager.cc:4830] TS 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): tablet 279b5f4d8fba4c12beec040c6ed39992 (table rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]) successfully deleted
I20240501 08:13:38.796872 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:38.812695 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:38.824712 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:38.838182 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:38.862208 12953 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 08:13:38.866457 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:38.910202 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:38.910236 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:39.069979 15754 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b"
dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:53732
I20240501 08:13:39.070156 15754 raft_consensus.cc:595] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:39.213466 27465 raft_consensus.cc:979] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: : Instructing follower 76d830e199eb461c8ca689c9460ee0c2 to start an election
I20240501 08:13:39.213596 27465 raft_consensus.cc:1067] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Signalling peer 76d830e199eb461c8ca689c9460ee0c2 to start an election
I20240501 08:13:39.215020 12971 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b"
dest_uuid: "76d830e199eb461c8ca689c9460ee0c2"
from {username='slave'} at 127.6.56.69:47087
I20240501 08:13:39.215334 12971 raft_consensus.cc:482] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:39.215456 12971 raft_consensus.cc:3037] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:39.230594 12971 raft_consensus.cc:504] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:39.232848 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:13:39.233055 15754 raft_consensus.cc:3032] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:39.233108 15754 raft_consensus.cc:729] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:13:39.233310 15754 consensus_queue.cc:260] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:39.233493 16024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:13:39.233714 15754 raft_consensus.cc:3037] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:39.233747 16024 raft_consensus.cc:3037] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:39.237169 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "76d830e199eb461c8ca689c9460ee0c2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:13:39.237371 16157 raft_consensus.cc:3037] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:39.237752 16024 raft_consensus.cc:2454] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 2.
I20240501 08:13:39.238817 12971 leader_election.cc:288] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 2 election: Requested vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:39.240243 16157 raft_consensus.cc:2454] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 2.
I20240501 08:13:39.240535 15754 raft_consensus.cc:2454] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76d830e199eb461c8ca689c9460ee0c2 in term 2.
I20240501 08:13:39.242363 12909 leader_election.cc:302] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:13:39.244199 27220 raft_consensus.cc:2781] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:39.246208 27220 raft_consensus.cc:686] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 2 LEADER]: Becoming Leader. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Running, Role: LEADER
I20240501 08:13:39.246575 27220 consensus_queue.cc:237] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:39.309170 12825 catalog_manager.cc:5484] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: term changed from 1 to 2, leader changed from 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) to 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65). New cstate: current_term: 2 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:39.428805 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
W20240501 08:13:39.486601 13301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 78415.9us
I20240501 08:13:39.486830 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:39.486860 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:39.496188 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:39.526657 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:39.549942 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:39.578477 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:13:39.583437 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 52736.2us
I20240501 08:13:39.607836 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:39.699678 16024 raft_consensus.cc:1261] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:39.701079 27732 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:39.727454 16157 raft_consensus.cc:1261] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:39.728593 27732 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:39.765276 15753 raft_consensus.cc:1261] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:39.767707 27732 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:40.172498 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:40.181490 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:40.204849 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:40.226657 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:40.234508 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:40.234905 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:40.310933 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:40.413264 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:40.655367 15753 consensus_queue.cc:237] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:40.657671 15892 raft_consensus.cc:1261] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:40.658437 12973 raft_consensus.cc:1261] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:40.658524 25695 consensus_queue.cc:1035] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:40.661221 27465 consensus_queue.cc:1035] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:40.662653 25695 raft_consensus.cc:2932] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:40.663259 15891 raft_consensus.cc:2932] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:40.664553 12973 raft_consensus.cc:2932] T 949d6765e68146d998ff4f5d2b120d68 P 76d830e199eb461c8ca689c9460ee0c2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:40.669993 12825 catalog_manager.cc:5484] T 949d6765e68146d998ff4f5d2b120d68 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 4, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) evicted. New cstate: current_term: 2 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:40.683981 13084 tablet_service.cc:1514] Processing DeleteTablet for tablet 949d6765e68146d998ff4f5d2b120d68 with delete_type TABLET_DATA_TOMBSTONED (TS e3fefd4dca2d41678187047f06c10187 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58510
I20240501 08:13:40.686740 27895 tablet_replica.cc:330] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: stopping tablet replica
I20240501 08:13:40.687008 27895 raft_consensus.cc:2229] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:40.687175 27895 raft_consensus.cc:2258] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:40.687913 27895 ts_tablet_manager.cc:1875] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:40.693441 27895 ts_tablet_manager.cc:1888] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 08:13:40.693675 27895 log.cc:1196] T 949d6765e68146d998ff4f5d2b120d68 P e3fefd4dca2d41678187047f06c10187: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/wals/949d6765e68146d998ff4f5d2b120d68
I20240501 08:13:40.694516 12819 catalog_manager.cc:4830] TS e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): tablet 949d6765e68146d998ff4f5d2b120d68 (table rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]) successfully deleted
I20240501 08:13:40.950642 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:40.971298 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:40.994614 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:41.004148 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:41.004514 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:41.053633 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:41.086141 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:41.138307 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:41.280764 16024 consensus_queue.cc:237] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:41.283200 13103 raft_consensus.cc:1261] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:41.283254 12971 raft_consensus.cc:1261] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:41.283774 27099 consensus_queue.cc:1035] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:41.284121 27221 consensus_queue.cc:1035] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:41.288797 27099 raft_consensus.cc:2932] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:41.289566 12971 raft_consensus.cc:2932] T ca6339e48a2f4b9e94e32e05c7bf343b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:41.290025 13103 raft_consensus.cc:2932] T ca6339e48a2f4b9e94e32e05c7bf343b P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:41.294548 12828 catalog_manager.cc:5484] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:41.308833 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet ca6339e48a2f4b9e94e32e05c7bf343b with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34524
I20240501 08:13:41.309649 27985 tablet_replica.cc:330] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:13:41.310154 27985 raft_consensus.cc:2229] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:41.310443 27985 raft_consensus.cc:2258] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:41.311326 27985 ts_tablet_manager.cc:1875] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:41.317963 27985 ts_tablet_manager.cc:1888] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:41.318531 27985 log.cc:1196] T ca6339e48a2f4b9e94e32e05c7bf343b P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/ca6339e48a2f4b9e94e32e05c7bf343b
I20240501 08:13:41.320398 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet ca6339e48a2f4b9e94e32e05c7bf343b (table rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]) successfully deleted
I20240501 08:13:41.418519 12971 consensus_queue.cc:237] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:41.421147 16158 raft_consensus.cc:1261] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:41.421298 15755 raft_consensus.cc:1261] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:41.421394 12909 consensus_peers.cc:485] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 714a84eae9324caeb4f25e798d6c35c2. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:41.421648 28024 consensus_queue.cc:1035] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.421958 27924 consensus_queue.cc:1035] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.424397 27924 raft_consensus.cc:2932] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:41.424955 16158 raft_consensus.cc:2932] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:41.428117 15755 raft_consensus.cc:2932] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:41.430887 12831 catalog_manager.cc:5484] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:41.466265 16024 consensus_queue.cc:237] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:41.468385 13105 raft_consensus.cc:1261] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:41.468665 16158 raft_consensus.cc:1261] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:41.469012 15959 consensus_peers.cc:485] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b320e884c36d461ba5f2941d9d76185f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:41.469143 27099 consensus_queue.cc:1035] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.469352 27221 consensus_queue.cc:1035] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.477242 27221 raft_consensus.cc:2932] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:41.478132 16158 raft_consensus.cc:2932] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:41.481818 13105 raft_consensus.cc:2932] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:41.484627 12827 catalog_manager.cc:5484] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:41.607913 13363 consensus_queue.cc:237] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:41.610606 16158 raft_consensus.cc:1261] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:41.610651 13301 consensus_peers.cc:485] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a713c94c6bb14d8793ee429e52f0e7a0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:41.610641 16024 raft_consensus.cc:1261] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:41.611089 27014 consensus_queue.cc:1035] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.611383 27959 consensus_queue.cc:1035] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.627811 27014 raft_consensus.cc:2932] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:41.628501 16158 raft_consensus.cc:2932] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:41.633217 16024 raft_consensus.cc:2932] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:41.637997 12831 catalog_manager.cc:5484] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:41.688774 16024 consensus_queue.cc:237] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:41.691336 15891 raft_consensus.cc:1261] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:41.691520 15960 consensus_peers.cc:485] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 315d149a26c14982aae10b5fb800dc82. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:41.691519 16158 raft_consensus.cc:1261] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:41.691812 28047 consensus_queue.cc:1035] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.692045 28046 consensus_queue.cc:1035] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.698551 28047 raft_consensus.cc:2932] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:41.699110 16158 raft_consensus.cc:2932] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:41.702904 15891 raft_consensus.cc:2932] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:41.705796 12827 catalog_manager.cc:5484] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:41.751106 15891 consensus_queue.cc:237] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:41.754001 13232 raft_consensus.cc:1261] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:41.754598 15755 raft_consensus.cc:1261] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:41.754568 28101 consensus_queue.cc:1035] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:41.755884 15823 consensus_peers.cc:485] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80ef1222e2324309a009a657ca704795. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:41.756554 28101 consensus_queue.cc:1035] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.759280 26452 raft_consensus.cc:2932] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:41.759857 15755 raft_consensus.cc:2932] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:41.766124 13232 raft_consensus.cc:2932] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:41.773581 12827 catalog_manager.cc:5484] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index -1 to 2, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:41.803805 15755 consensus_queue.cc:237] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:41.806317 16024 raft_consensus.cc:1261] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:41.806352 15691 consensus_peers.cc:485] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 691f5a8c1d334f6ba6464c9596db49ed. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:41.806530 13363 raft_consensus.cc:1261] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:41.807111 25695 consensus_queue.cc:1035] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.807632 25695 consensus_queue.cc:1035] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.811851 25695 raft_consensus.cc:2932] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:41.813232 16023 raft_consensus.cc:2932] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:41.822672 12831 catalog_manager.cc:5484] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:41.834163 13363 raft_consensus.cc:2932] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:41.861892 16157 consensus_queue.cc:237] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:41.864250 16023 raft_consensus.cc:1261] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:41.864403 16092 consensus_peers.cc:485] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ba04c9b0d0a4933a123dbeb5b50145e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:41.864790 28141 consensus_queue.cc:1035] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.864912 12971 raft_consensus.cc:1261] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:41.865459 28141 consensus_queue.cc:1035] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.868114 27761 raft_consensus.cc:2932] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:41.868683 12971 raft_consensus.cc:2932] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:41.875941 12831 catalog_manager.cc:5484] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:41.883942 16023 raft_consensus.cc:2932] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:41.961529 28160 ts_tablet_manager.cc:902] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:13:41.965904 15755 consensus_queue.cc:237] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:41.968428 13105 raft_consensus.cc:1261] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:41.968662 12971 raft_consensus.cc:1261] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:41.968701 15692 consensus_peers.cc:485] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6840d246ddfa42b3b8065d7136a36f5a. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:41.969133 28128 consensus_queue.cc:1035] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.969668 28128 consensus_queue.cc:1035] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:41.981514 28160 tablet_copy_client.cc:308] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:13:41.984232 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet 714a84eae9324caeb4f25e798d6c35c2 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:60299)
I20240501 08:13:41.984448 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet 714a84eae9324caeb4f25e798d6c35c2 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:60299: session id = 4ecb724cb4764a53988fde00407db4c0-714a84eae9324caeb4f25e798d6c35c2
I20240501 08:13:41.987704 12994 tablet_copy_source_session.cc:205] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:41.989446 28160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 714a84eae9324caeb4f25e798d6c35c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:41.991329 27465 raft_consensus.cc:2932] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:41.992669 13105 raft_consensus.cc:2932] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:41.999123 28160 tablet_copy_client.cc:783] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:41.999329 28160 tablet_copy_client.cc:647] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:42.000835 12828 catalog_manager.cc:5484] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:42.002000 28160 tablet_copy_client.cc:523] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:42.002390 12971 raft_consensus.cc:2932] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:42.005777 28160 tablet_bootstrap.cc:492] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:42.012380 28160 tablet_bootstrap.cc:492] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:42.012777 28160 tablet_bootstrap.cc:492] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:42.013041 28160 ts_tablet_manager.cc:1367] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240501 08:13:42.013880 28160 raft_consensus.cc:348] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:42.014123 28160 raft_consensus.cc:729] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:42.014438 28160 consensus_queue.cc:260] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:42.015466 28160 ts_tablet_manager.cc:1398] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:42.016427 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-714a84eae9324caeb4f25e798d6c35c2 received from {username='slave'} at 127.6.56.68:60299
I20240501 08:13:42.016572 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-714a84eae9324caeb4f25e798d6c35c2 on tablet 714a84eae9324caeb4f25e798d6c35c2 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:42.069743 16023 consensus_queue.cc:237] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
W20240501 08:13:42.073700 15959 consensus_peers.cc:485] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4345eabfdb894def8d7063773dc5fac6. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:42.073877 12971 raft_consensus.cc:1261] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:42.074579 27221 consensus_queue.cc:1035] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:42.074807 16157 raft_consensus.cc:1261] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:42.075937 27221 consensus_queue.cc:1035] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:42.079948 28046 raft_consensus.cc:2932] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:42.080612 16158 raft_consensus.cc:2932] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:42.089208 12973 raft_consensus.cc:2932] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:42.090955 28185 ts_tablet_manager.cc:902] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:42.094264 12831 catalog_manager.cc:5484] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:42.112159 28185 tablet_copy_client.cc:308] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:13:42.113086 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet b320e884c36d461ba5f2941d9d76185f from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:48147)
I20240501 08:13:42.113258 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet b320e884c36d461ba5f2941d9d76185f from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:48147: session id = 982f49fc87f149439cda0de2261b6bfd-b320e884c36d461ba5f2941d9d76185f
I20240501 08:13:42.115562 16045 tablet_copy_source_session.cc:205] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:42.116920 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b320e884c36d461ba5f2941d9d76185f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:42.124912 28185 tablet_copy_client.cc:783] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:42.125197 28185 tablet_copy_client.cc:647] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:42.129323 28185 tablet_copy_client.cc:523] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:42.132246 28185 tablet_bootstrap.cc:492] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:13:42.140203 28185 tablet_bootstrap.cc:492] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:42.140727 28185 tablet_bootstrap.cc:492] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:13:42.141911 28185 ts_tablet_manager.cc:1367] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.004s
I20240501 08:13:42.142714 28185 raft_consensus.cc:348] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:42.142903 28185 raft_consensus.cc:729] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:13:42.143121 28185 consensus_queue.cc:260] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:42.143815 28185 ts_tablet_manager.cc:1398] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:13:42.144555 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-b320e884c36d461ba5f2941d9d76185f received from {username='slave'} at 127.6.56.69:48147
I20240501 08:13:42.144680 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-b320e884c36d461ba5f2941d9d76185f on tablet b320e884c36d461ba5f2941d9d76185f with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:13:42.180881 28204 ts_tablet_manager.cc:902] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:42.182205 28204 tablet_copy_client.cc:308] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:42.182898 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet a713c94c6bb14d8793ee429e52f0e7a0 from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:58107)
I20240501 08:13:42.183104 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet a713c94c6bb14d8793ee429e52f0e7a0 from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:58107: session id = 6ed448d04e4646658c0de37d829ff15b-a713c94c6bb14d8793ee429e52f0e7a0
I20240501 08:13:42.185714 13385 tablet_copy_source_session.cc:205] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:42.187136 28204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a713c94c6bb14d8793ee429e52f0e7a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:42.192272 28204 tablet_copy_client.cc:783] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:42.192516 28204 tablet_copy_client.cc:647] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:42.195288 28204 tablet_copy_client.cc:523] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:42.197495 28204 tablet_bootstrap.cc:492] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:13:42.202553 28204 tablet_bootstrap.cc:492] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:42.202888 28204 tablet_bootstrap.cc:492] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:13:42.203094 28204 ts_tablet_manager.cc:1367] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 08:13:42.203658 28204 raft_consensus.cc:348] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:42.203827 28204 raft_consensus.cc:729] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:13:42.204037 28204 consensus_queue.cc:260] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:42.204509 28204 ts_tablet_manager.cc:1398] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 08:13:42.205108 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-a713c94c6bb14d8793ee429e52f0e7a0 received from {username='slave'} at 127.6.56.67:58107
I20240501 08:13:42.205245 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-a713c94c6bb14d8793ee429e52f0e7a0 on tablet a713c94c6bb14d8793ee429e52f0e7a0 with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:13:42.252676 28227 ts_tablet_manager.cc:902] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:42.276000 28227 tablet_copy_client.cc:308] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:13:42.276688 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 315d149a26c14982aae10b5fb800dc82 from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:52501)
I20240501 08:13:42.276862 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 315d149a26c14982aae10b5fb800dc82 from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:52501: session id = 76d830e199eb461c8ca689c9460ee0c2-315d149a26c14982aae10b5fb800dc82
I20240501 08:13:42.279294 16045 tablet_copy_source_session.cc:205] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:42.280467 28227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 315d149a26c14982aae10b5fb800dc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:42.291960 28227 tablet_copy_client.cc:783] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:42.292251 28227 tablet_copy_client.cc:647] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:42.294307 28238 ts_tablet_manager.cc:902] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:13:42.297487 28227 tablet_copy_client.cc:523] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:42.303025 28227 tablet_bootstrap.cc:492] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:42.314507 28238 tablet_copy_client.cc:308] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:13:42.315371 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 80ef1222e2324309a009a657ca704795 from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:34463)
I20240501 08:13:42.315577 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 80ef1222e2324309a009a657ca704795 from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:34463: session id = 6cc32f6e2d49477f9a23ce6e444039fb-80ef1222e2324309a009a657ca704795
I20240501 08:13:42.318846 15913 tablet_copy_source_session.cc:205] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:42.319123 28227 tablet_bootstrap.cc:492] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:42.319569 28227 tablet_bootstrap.cc:492] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:42.319849 28227 ts_tablet_manager.cc:1367] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20240501 08:13:42.320657 28227 raft_consensus.cc:348] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:42.320909 28227 raft_consensus.cc:729] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:42.321182 28227 consensus_queue.cc:260] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:42.321789 28227 ts_tablet_manager.cc:1398] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240501 08:13:42.322472 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-315d149a26c14982aae10b5fb800dc82 received from {username='slave'} at 127.6.56.65:52501
I20240501 08:13:42.322618 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-315d149a26c14982aae10b5fb800dc82 on tablet 315d149a26c14982aae10b5fb800dc82 with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:13:42.324210 28238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ef1222e2324309a009a657ca704795. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:42.334208 28160 ts_tablet_manager.cc:902] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:13:42.341905 28251 ts_tablet_manager.cc:902] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:13:42.346590 28238 tablet_copy_client.cc:783] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:42.346999 28238 tablet_copy_client.cc:647] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:42.349670 28238 tablet_copy_client.cc:523] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:42.354444 28238 tablet_bootstrap.cc:492] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:13:42.355576 28251 tablet_copy_client.cc:308] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:13:42.356420 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet 691f5a8c1d334f6ba6464c9596db49ed from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:33955)
I20240501 08:13:42.356650 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet 691f5a8c1d334f6ba6464c9596db49ed from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:33955: session id = e3fefd4dca2d41678187047f06c10187-691f5a8c1d334f6ba6464c9596db49ed
I20240501 08:13:42.359719 15776 tablet_copy_source_session.cc:205] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:42.361114 28251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 691f5a8c1d334f6ba6464c9596db49ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:42.366953 28251 tablet_copy_client.cc:783] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:42.367180 28251 tablet_copy_client.cc:647] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:42.372929 28251 tablet_copy_client.cc:523] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:42.378916 28251 tablet_bootstrap.cc:492] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:13:42.383575 28238 tablet_bootstrap.cc:492] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:42.384081 28238 tablet_bootstrap.cc:492] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:13:42.395035 28238 ts_tablet_manager.cc:1367] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.002s
I20240501 08:13:42.396335 28160 tablet_copy_client.cc:308] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.72:45383
I20240501 08:13:42.397154 16178 tablet_copy_service.cc:140] P 6cc32f6e2d49477f9a23ce6e444039fb: Received BeginTabletCopySession request for tablet 1ba04c9b0d0a4933a123dbeb5b50145e from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:52203)
I20240501 08:13:42.397333 16178 tablet_copy_service.cc:161] P 6cc32f6e2d49477f9a23ce6e444039fb: Beginning new tablet copy session on tablet 1ba04c9b0d0a4933a123dbeb5b50145e from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:52203: session id = 4ecb724cb4764a53988fde00407db4c0-1ba04c9b0d0a4933a123dbeb5b50145e
I20240501 08:13:42.401234 28251 tablet_bootstrap.cc:492] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:42.401754 28251 tablet_bootstrap.cc:492] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:13:42.401883 16178 tablet_copy_source_session.cc:205] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:42.402009 28251 ts_tablet_manager.cc:1367] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.001s
I20240501 08:13:42.402714 28251 raft_consensus.cc:348] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:42.402962 28251 raft_consensus.cc:729] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:13:42.403632 28160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba04c9b0d0a4933a123dbeb5b50145e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:42.403530 28251 consensus_queue.cc:260] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:42.404325 28251 ts_tablet_manager.cc:1398] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:42.407195 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-691f5a8c1d334f6ba6464c9596db49ed received from {username='slave'} at 127.6.56.66:33955
I20240501 08:13:42.407363 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session e3fefd4dca2d41678187047f06c10187-691f5a8c1d334f6ba6464c9596db49ed on tablet 691f5a8c1d334f6ba6464c9596db49ed with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:13:42.395854 28238 raft_consensus.cc:348] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:42.418215 28238 raft_consensus.cc:729] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:13:42.418663 28238 consensus_queue.cc:260] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:42.420379 13363 raft_consensus.cc:1203] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:42.423851 28238 ts_tablet_manager.cc:1398] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20240501 08:13:42.426443 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-80ef1222e2324309a009a657ca704795 received from {username='slave'} at 127.6.56.72:34463
I20240501 08:13:42.426553 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-80ef1222e2324309a009a657ca704795 on tablet 80ef1222e2324309a009a657ca704795 with peer 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:13:42.453073 28160 tablet_copy_client.cc:783] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:42.453469 28160 tablet_copy_client.cc:647] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:42.455891 28160 tablet_copy_client.cc:523] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:42.459477 28160 tablet_bootstrap.cc:492] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:42.484432 28160 tablet_bootstrap.cc:492] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:42.484875 28160 tablet_bootstrap.cc:492] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:42.485167 28160 ts_tablet_manager.cc:1367] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.004s
I20240501 08:13:42.486066 28160 raft_consensus.cc:348] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:42.486321 28160 raft_consensus.cc:729] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:42.486541 28160 consensus_queue.cc:260] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:42.487509 28160 ts_tablet_manager.cc:1398] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:42.488517 16178 tablet_copy_service.cc:342] P 6cc32f6e2d49477f9a23ce6e444039fb: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-1ba04c9b0d0a4933a123dbeb5b50145e received from {username='slave'} at 127.6.56.68:52203
I20240501 08:13:42.488659 16178 tablet_copy_service.cc:433] P 6cc32f6e2d49477f9a23ce6e444039fb: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-1ba04c9b0d0a4933a123dbeb5b50145e on tablet 1ba04c9b0d0a4933a123dbeb5b50145e with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:42.509536 28160 ts_tablet_manager.cc:902] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:42.569281 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:42.575323 28160 tablet_copy_client.cc:308] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:13:42.576038 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet 4345eabfdb894def8d7063773dc5fac6 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:50471)
I20240501 08:13:42.576190 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet 4345eabfdb894def8d7063773dc5fac6 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:50471: session id = 4ecb724cb4764a53988fde00407db4c0-4345eabfdb894def8d7063773dc5fac6
I20240501 08:13:42.579145 16045 tablet_copy_source_session.cc:205] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:42.580437 28160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4345eabfdb894def8d7063773dc5fac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:42.582938 28204 ts_tablet_manager.cc:902] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:13:42.591621 28160 tablet_copy_client.cc:783] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:42.591883 28160 tablet_copy_client.cc:647] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:42.591976 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:42.594077 28160 tablet_copy_client.cc:523] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:42.607832 28160 tablet_bootstrap.cc:492] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:42.630764 15755 raft_consensus.cc:1203] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:42.634552 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:42.642112 28160 tablet_bootstrap.cc:492] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:42.642514 28160 tablet_bootstrap.cc:492] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:42.642743 28160 ts_tablet_manager.cc:1367] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.003s
I20240501 08:13:42.643457 28160 raft_consensus.cc:348] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:42.643658 28160 raft_consensus.cc:729] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:42.643848 28160 consensus_queue.cc:260] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:42.646394 28160 ts_tablet_manager.cc:1398] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240501 08:13:42.647099 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-4345eabfdb894def8d7063773dc5fac6 received from {username='slave'} at 127.6.56.68:50471
I20240501 08:13:42.647215 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-4345eabfdb894def8d7063773dc5fac6 on tablet 4345eabfdb894def8d7063773dc5fac6 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:42.683012 12971 raft_consensus.cc:1203] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:42.695464 28204 tablet_copy_client.cc:308] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:13:42.696218 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet 6840d246ddfa42b3b8065d7136a36f5a from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:49819)
I20240501 08:13:42.696455 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet 6840d246ddfa42b3b8065d7136a36f5a from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:49819: session id = 6ed448d04e4646658c0de37d829ff15b-6840d246ddfa42b3b8065d7136a36f5a
I20240501 08:13:42.699962 15776 tablet_copy_source_session.cc:205] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:42.701512 28204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6840d246ddfa42b3b8065d7136a36f5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:42.721812 28204 tablet_copy_client.cc:783] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:42.722100 28204 tablet_copy_client.cc:647] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:42.725490 28204 tablet_copy_client.cc:523] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:42.743899 28204 tablet_bootstrap.cc:492] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:13:42.759294 13232 raft_consensus.cc:1203] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:42.760597 28204 tablet_bootstrap.cc:492] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:42.760991 28204 tablet_bootstrap.cc:492] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:13:42.761317 28204 ts_tablet_manager.cc:1367] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240501 08:13:42.761992 28204 raft_consensus.cc:348] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:42.762228 28204 raft_consensus.cc:729] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:13:42.762522 28204 consensus_queue.cc:260] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:42.763170 28204 ts_tablet_manager.cc:1398] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:13:42.764245 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-6840d246ddfa42b3b8065d7136a36f5a received from {username='slave'} at 127.6.56.67:49819
I20240501 08:13:42.764407 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-6840d246ddfa42b3b8065d7136a36f5a on tablet 6840d246ddfa42b3b8065d7136a36f5a with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:13:42.805346 13364 raft_consensus.cc:1203] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:42.807768 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:42.860738 16157 raft_consensus.cc:1203] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:42.868306 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:42.870133 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:42.894840 13364 raft_consensus.cc:1203] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:42.898367 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:42.906388 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:42.926155 13105 raft_consensus.cc:1203] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:43.183012 13232 raft_consensus.cc:1203] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:43.323302 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:43.374007 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:43.399942 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:43.480924 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:43.487313 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:43.495661 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:43.500108 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:43.540846 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:43.707353 15755 consensus_queue.cc:237] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:43.709499 13232 raft_consensus.cc:1261] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:43.710072 27465 consensus_queue.cc:1035] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:43.710173 13104 raft_consensus.cc:1261] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:43.711062 27465 consensus_queue.cc:1035] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:43.712909 28121 raft_consensus.cc:2932] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:43.714272 13104 raft_consensus.cc:2932] T 6840d246ddfa42b3b8065d7136a36f5a P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:43.716871 13232 raft_consensus.cc:2932] T 6840d246ddfa42b3b8065d7136a36f5a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:43.719053 12832 catalog_manager.cc:5484] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:43.741608 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet 6840d246ddfa42b3b8065d7136a36f5a with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:57838
I20240501 08:13:43.755805 28393 tablet_replica.cc:330] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:13:43.756050 28393 raft_consensus.cc:2229] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:43.756275 28393 raft_consensus.cc:2258] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:43.757712 28393 ts_tablet_manager.cc:1875] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:43.766709 28393 ts_tablet_manager.cc:1888] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:43.766863 28393 log.cc:1196] T 6840d246ddfa42b3b8065d7136a36f5a P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/6840d246ddfa42b3b8065d7136a36f5a
I20240501 08:13:43.767590 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet 6840d246ddfa42b3b8065d7136a36f5a (table rebalancer_extra_table_20 [id=3cf0264a63a549bdb76fec4f0b314d0e]) successfully deleted
I20240501 08:13:44.012979 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:44.021274 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:44.104568 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:44.146286 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:44.150498 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:44.188089 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:44.224579 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:44.263810 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:44.267838 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(a54efcd41aeb4c24adfa51bb543d9661): perf score=0.033339
I20240501 08:13:44.318693 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(4fc5dbfe3a9b4c84a3217981527e73ea): perf score=0.033358
I20240501 08:13:44.397639 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(99ecb2ec0b3d4fd7a02896cc9a28ba27): perf score=0.033375
I20240501 08:13:44.407204 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(a54efcd41aeb4c24adfa51bb543d9661) complete. Timing: real 0.139s user 0.085s sys 0.047s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":5510,"dirs.queue_time_us":59260,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":3042,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":888,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":71404,"threads_started":3,"wal-append.queue_time_us":6610}
I20240501 08:13:44.410867 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(a54efcd41aeb4c24adfa51bb543d9661): 4921 bytes on disk
I20240501 08:13:44.421178 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(a54efcd41aeb4c24adfa51bb543d9661) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
W20240501 08:13:44.470224 13170 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.69:42897, user_credentials={real_user=slave}} blocked reactor thread for 82868.6us
I20240501 08:13:44.483613 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(a54efcd41aeb4c24adfa51bb543d9661): perf score=0.033402
I20240501 08:13:44.573549 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(99ecb2ec0b3d4fd7a02896cc9a28ba27) complete. Timing: real 0.175s user 0.082s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":53750,"dirs.queue_time_us":6324,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":8444,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":823,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":65899,"threads_started":3,"wal-append.queue_time_us":2563}
I20240501 08:13:44.576993 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(99ecb2ec0b3d4fd7a02896cc9a28ba27): 4581 bytes on disk
I20240501 08:13:44.586376 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(99ecb2ec0b3d4fd7a02896cc9a28ba27) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 08:13:44.589191 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(4fc5dbfe3a9b4c84a3217981527e73ea): perf score=0.033426
I20240501 08:13:44.770498 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(4fc5dbfe3a9b4c84a3217981527e73ea) complete. Timing: real 0.181s user 0.054s sys 0.044s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":846,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":79407,"threads_started":1,"wal-append.queue_time_us":79301}
I20240501 08:13:44.773394 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(4fc5dbfe3a9b4c84a3217981527e73ea): 4834 bytes on disk
I20240501 08:13:44.774035 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(4fc5dbfe3a9b4c84a3217981527e73ea) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 08:13:44.851725 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(4fc5dbfe3a9b4c84a3217981527e73ea) complete. Timing: real 0.532s user 0.074s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":14896,"dirs.queue_time_us":9786,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":923,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1006,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":79724,"threads_started":3,"wal-append.queue_time_us":20900}
I20240501 08:13:44.856824 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(4fc5dbfe3a9b4c84a3217981527e73ea): 4834 bytes on disk
I20240501 08:13:44.865285 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(4fc5dbfe3a9b4c84a3217981527e73ea) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
W20240501 08:13:45.057863 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 215571us
I20240501 08:13:45.179303 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(a54efcd41aeb4c24adfa51bb543d9661) complete. Timing: real 0.695s user 0.092s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":6650,"dirs.queue_time_us":14632,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1070,"drs_written":1,"lbm_read_time_us":149,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4061,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":217962,"threads_started":3,"wal-append.queue_time_us":21413}
I20240501 08:13:45.209206 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(a54efcd41aeb4c24adfa51bb543d9661): 4921 bytes on disk
I20240501 08:13:45.210323 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(a54efcd41aeb4c24adfa51bb543d9661) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4}
I20240501 08:13:45.227982 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(99ecb2ec0b3d4fd7a02896cc9a28ba27): perf score=0.033605
I20240501 08:13:45.274220 16158 consensus_queue.cc:237] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:45.293318 13363 raft_consensus.cc:1261] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:45.294010 28340 consensus_queue.cc:1035] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:45.302279 12973 raft_consensus.cc:1261] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:45.302980 28340 consensus_queue.cc:1035] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:45.311391 28481 raft_consensus.cc:2932] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:45.312278 13363 raft_consensus.cc:2932] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:45.322636 12829 catalog_manager.cc:5484] T 1ba04c9b0d0a4933a123dbeb5b50145e P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:45.337044 12971 raft_consensus.cc:2932] T 1ba04c9b0d0a4933a123dbeb5b50145e P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:45.356148 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba04c9b0d0a4933a123dbeb5b50145e with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50326
I20240501 08:13:45.356760 28509 tablet_replica.cc:330] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:13:45.356976 28509 raft_consensus.cc:2229] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:45.357260 28509 raft_consensus.cc:2258] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:45.358078 28509 ts_tablet_manager.cc:1875] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:45.375105 28509 ts_tablet_manager.cc:1888] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:45.375353 28509 log.cc:1196] T 1ba04c9b0d0a4933a123dbeb5b50145e P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/1ba04c9b0d0a4933a123dbeb5b50145e
I20240501 08:13:45.376968 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet 1ba04c9b0d0a4933a123dbeb5b50145e (table rebalancer_extra_table_44 [id=b583638d0eca49b69d946c417d3533e9]) successfully deleted
I20240501 08:13:45.392973 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(99ecb2ec0b3d4fd7a02896cc9a28ba27) complete. Timing: real 0.162s user 0.051s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8809,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":13566,"drs_written":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4,"lbm_write_time_us":20365,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":15555,"threads_started":2,"wal-append.queue_time_us":6367}
I20240501 08:13:45.395758 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(99ecb2ec0b3d4fd7a02896cc9a28ba27): 4581 bytes on disk
I20240501 08:13:45.396809 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(99ecb2ec0b3d4fd7a02896cc9a28ba27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240501 08:13:45.592692 15825 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 45117us
I20240501 08:13:46.384922 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:46.406934 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:46.475018 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:46.497856 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:46.522346 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240501 08:13:46.583317 16090 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 134866us
I20240501 08:13:46.649201 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:46.657052 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:46.663004 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:46.798785 15755 consensus_queue.cc:237] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:13:46.801008 13104 raft_consensus.cc:1261] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:46.801631 28487 consensus_queue.cc:1035] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:46.801885 16023 raft_consensus.cc:1261] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:46.802842 28487 consensus_queue.cc:1035] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:46.805547 28471 raft_consensus.cc:2932] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:46.806262 13104 raft_consensus.cc:2932] T 691f5a8c1d334f6ba6464c9596db49ed P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:46.807946 16023 raft_consensus.cc:2932] T 691f5a8c1d334f6ba6464c9596db49ed P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:46.812927 12824 catalog_manager.cc:5484] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:13:46.819738 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet 691f5a8c1d334f6ba6464c9596db49ed with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56180
I20240501 08:13:46.820348 28605 tablet_replica.cc:330] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:13:46.820556 28605 raft_consensus.cc:2229] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:46.820746 28605 raft_consensus.cc:2258] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:46.821321 28605 ts_tablet_manager.cc:1875] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:46.826400 28605 ts_tablet_manager.cc:1888] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:46.826571 28605 log.cc:1196] T 691f5a8c1d334f6ba6464c9596db49ed P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/691f5a8c1d334f6ba6464c9596db49ed
I20240501 08:13:46.827381 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 691f5a8c1d334f6ba6464c9596db49ed (table rebalancer_extra_table_42 [id=296e1e244a9f4ff9b0d6f616b200011f]) successfully deleted
I20240501 08:13:47.148455 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:47.153259 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:47.189728 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:47.227427 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:47.243147 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:47.257687 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:47.293124 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:47.365799 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:47.463281 15892 consensus_queue.cc:237] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:47.465055 16158 raft_consensus.cc:1261] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:47.465077 13232 raft_consensus.cc:1261] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:47.465538 28666 consensus_queue.cc:1035] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:47.465801 28579 consensus_queue.cc:1035] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:47.468389 28579 raft_consensus.cc:2932] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:47.471709 16154 raft_consensus.cc:2932] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:47.477586 12828 catalog_manager.cc:5484] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:47.488106 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ef1222e2324309a009a657ca704795 with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:40166
I20240501 08:13:47.497541 13232 raft_consensus.cc:2932] T 80ef1222e2324309a009a657ca704795 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:47.504137 28691 tablet_replica.cc:330] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:13:47.504428 28691 raft_consensus.cc:2229] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:47.504698 28691 raft_consensus.cc:2258] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:47.505506 28691 ts_tablet_manager.cc:1875] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:47.514171 28691 ts_tablet_manager.cc:1888] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:47.514416 28691 log.cc:1196] T 80ef1222e2324309a009a657ca704795 P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/80ef1222e2324309a009a657ca704795
I20240501 08:13:47.515779 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet 80ef1222e2324309a009a657ca704795 (table rebalancer_extra_table_46 [id=c49cff7be46147bb944476c4c8cf9d29]) successfully deleted
I20240501 08:13:47.781184 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:47.785231 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:47.809032 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:47.838721 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:47.846400 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:47.866561 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:47.942473 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:47.952389 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:48.101038 16024 consensus_queue.cc:237] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:48.103812 12973 raft_consensus.cc:1261] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:48.103788 15891 raft_consensus.cc:1261] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:48.104279 28750 consensus_queue.cc:1035] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:48.104565 28751 consensus_queue.cc:1035] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:48.107607 28751 raft_consensus.cc:2932] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:48.108024 15891 raft_consensus.cc:2932] T 315d149a26c14982aae10b5fb800dc82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:48.108338 12973 raft_consensus.cc:2932] T 315d149a26c14982aae10b5fb800dc82 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:48.114087 12832 catalog_manager.cc:5484] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:48.120733 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 315d149a26c14982aae10b5fb800dc82 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:13:48.121299 28782 tablet_replica.cc:330] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:13:48.121490 28782 raft_consensus.cc:2229] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:48.121680 28782 raft_consensus.cc:2258] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:48.122277 28782 ts_tablet_manager.cc:1875] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:48.128547 28782 ts_tablet_manager.cc:1888] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:48.128706 28782 log.cc:1196] T 315d149a26c14982aae10b5fb800dc82 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/315d149a26c14982aae10b5fb800dc82
I20240501 08:13:48.129628 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 315d149a26c14982aae10b5fb800dc82 (table rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]) successfully deleted
I20240501 08:13:48.316928 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(31b3831e0f36425982b1627dd5508c99): perf score=0.033353
I20240501 08:13:48.409860 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(31b3831e0f36425982b1627dd5508c99): perf score=0.033374
I20240501 08:13:48.426460 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(31b3831e0f36425982b1627dd5508c99): perf score=0.033380
I20240501 08:13:48.515875 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:48.521003 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:48.523806 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(7a98388bf2fd402b8064d2cca06f5b8b): perf score=0.033403
I20240501 08:13:48.530766 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:48.539594 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:48.555652 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:48.612067 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:48.613426 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(31b3831e0f36425982b1627dd5508c99) complete. Timing: real 0.296s user 0.073s sys 0.048s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":33813,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":5334,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":853,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":76739,"threads_started":2,"wal-append.queue_time_us":42056}
I20240501 08:13:48.624366 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(31b3831e0f36425982b1627dd5508c99): 4921 bytes on disk
I20240501 08:13:48.626320 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(31b3831e0f36425982b1627dd5508c99) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 08:13:48.632380 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(74d6cce81d164722afad1935d4695374): perf score=0.033434
I20240501 08:13:48.667526 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:48.722256 13213 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:48.720520 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(31b3831e0f36425982b1627dd5508c99) complete. Timing: real 0.294s user 0.075s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":57078,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":9238,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5081,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":66591,"threads_started":2,"wal-append.queue_time_us":9439}
I20240501 08:13:48.730578 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(31b3831e0f36425982b1627dd5508c99): 4921 bytes on disk
I20240501 08:13:48.732090 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(31b3831e0f36425982b1627dd5508c99) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 08:13:48.763671 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(74d6cce81d164722afad1935d4695374) complete. Timing: real 0.131s user 0.037s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1364,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4519,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7611,"threads_started":1,"wal-append.queue_time_us":7721}
I20240501 08:13:48.769536 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(74d6cce81d164722afad1935d4695374): 4582 bytes on disk
I20240501 08:13:48.770251 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(74d6cce81d164722afad1935d4695374) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20240501 08:13:48.863773 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(7a98388bf2fd402b8064d2cca06f5b8b) complete. Timing: real 0.340s user 0.103s sys 0.055s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":106202,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":3648,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":782,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":106388,"threads_started":2,"wal-append.queue_time_us":386}
I20240501 08:13:48.867089 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(7a98388bf2fd402b8064d2cca06f5b8b): 4834 bytes on disk
I20240501 08:13:48.867794 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(7a98388bf2fd402b8064d2cca06f5b8b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240501 08:13:48.870606 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(74d6cce81d164722afad1935d4695374): perf score=0.033497
I20240501 08:13:48.881649 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(31b3831e0f36425982b1627dd5508c99) complete. Timing: real 0.472s user 0.047s sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":5944,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1325,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":934,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":896,"thread_start_us":9465,"threads_started":2,"wal-append.queue_time_us":3219}
I20240501 08:13:48.883953 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(31b3831e0f36425982b1627dd5508c99): 4921 bytes on disk
I20240501 08:13:48.886080 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(31b3831e0f36425982b1627dd5508c99) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 08:13:48.888207 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(7a98388bf2fd402b8064d2cca06f5b8b): perf score=0.033502
I20240501 08:13:48.947870 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(74d6cce81d164722afad1935d4695374) complete. Timing: real 0.077s user 0.042s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":135,"dirs.run_cpu_time_us":159,"dirs.run_wall_time_us":1410,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":911,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":26796,"threads_started":1,"wal-append.queue_time_us":26504}
I20240501 08:13:48.950707 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(74d6cce81d164722afad1935d4695374): 4582 bytes on disk
I20240501 08:13:48.951313 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(74d6cce81d164722afad1935d4695374) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 08:13:48.953969 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(7a98388bf2fd402b8064d2cca06f5b8b) complete. Timing: real 0.065s user 0.048s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1483,"drs_written":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1413,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":1408,"thread_start_us":3328,"threads_started":1,"wal-append.queue_time_us":2053}
I20240501 08:13:48.958345 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(7a98388bf2fd402b8064d2cca06f5b8b): 4834 bytes on disk
I20240501 08:13:48.958884 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(7a98388bf2fd402b8064d2cca06f5b8b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240501 08:13:49.004428 13364 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0"
dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40272
I20240501 08:13:49.004698 13364 raft_consensus.cc:595] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:49.254701 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:49.283550 13343 tablet_service.cc:1429] Tablet server has 20 leaders and 0 scanners
I20240501 08:13:49.299602 28489 raft_consensus.cc:979] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: : Instructing follower 6ed448d04e4646658c0de37d829ff15b to start an election
I20240501 08:13:49.299785 28489 raft_consensus.cc:1067] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Signalling peer 6ed448d04e4646658c0de37d829ff15b to start an election
I20240501 08:13:49.300595 13232 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0"
dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
from {username='slave'} at 127.6.56.68:56589
I20240501 08:13:49.300791 13232 raft_consensus.cc:482] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:49.300882 13232 raft_consensus.cc:3037] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:49.303532 13232 raft_consensus.cc:504] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:49.316278 13232 leader_election.cc:288] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: Requested vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:13:49.316612 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:13:49.316668 16157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:13:49.316819 16023 raft_consensus.cc:3037] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:49.316864 16157 raft_consensus.cc:3037] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:49.317031 13364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:13:49.317231 13364 raft_consensus.cc:3032] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:49.317297 13364 raft_consensus.cc:729] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:13:49.317613 13364 consensus_queue.cc:260] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:49.318151 13364 raft_consensus.cc:3037] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:49.319615 16023 raft_consensus.cc:2454] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 2.
I20240501 08:13:49.319754 16157 raft_consensus.cc:2454] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 2.
I20240501 08:13:49.320451 13170 leader_election.cc:302] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:13:49.320739 28659 raft_consensus.cc:2781] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:49.320901 28659 raft_consensus.cc:686] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [term 2 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:13:49.320924 13364 raft_consensus.cc:2454] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 2.
I20240501 08:13:49.321244 28659 consensus_queue.cc:237] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:49.346299 12833 catalog_manager.cc:5484] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 1 to 2, leader changed from 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 2 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:49.384054 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
W20240501 08:13:49.417836 15960 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 55558.3us
I20240501 08:13:49.434636 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:49.493322 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:49.493779 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:49.494509 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:49.550660 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:49.777396 16023 raft_consensus.cc:1261] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:49.778412 28659 consensus_queue.cc:1035] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:49.779338 13364 raft_consensus.cc:1261] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:49.780452 28659 consensus_queue.cc:1035] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:49.811785 16157 raft_consensus.cc:1261] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:49.812552 28458 consensus_queue.cc:1035] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:49.947240 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:49.988361 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:49.996327 16005 tablet_service.cc:1429] Tablet server has 22 leaders and 0 scanners
I20240501 08:13:50.064177 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:50.086998 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:50.126031 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:50.161268 15736 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:13:50.204568 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:50.302018 16023 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f"
dest_uuid: "5dd722c770d34d6c9154a87917117161"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34332
I20240501 08:13:50.302235 16023 raft_consensus.cc:595] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Received request to transfer leadership
I20240501 08:13:50.344725 28919 raft_consensus.cc:979] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: : Instructing follower 982f49fc87f149439cda0de2261b6bfd to start an election
I20240501 08:13:50.344898 28919 raft_consensus.cc:1067] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Signalling peer 982f49fc87f149439cda0de2261b6bfd to start an election
I20240501 08:13:50.346120 15753 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f"
dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
from {username='slave'} at 127.6.56.71:48491
I20240501 08:13:50.346349 15753 raft_consensus.cc:482] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:13:50.346443 15753 raft_consensus.cc:3037] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:50.350333 15753 raft_consensus.cc:504] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:50.351615 15753 leader_election.cc:288] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 election: Requested vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:13:50.352044 13104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:13:50.352344 13104 raft_consensus.cc:3037] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:50.355598 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:13:50.355880 16154 raft_consensus.cc:3037] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:50.359432 16154 raft_consensus.cc:2454] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 2.
I20240501 08:13:50.359933 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b320e884c36d461ba5f2941d9d76185f" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:13:50.360183 16023 raft_consensus.cc:3032] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:13:50.360266 16023 raft_consensus.cc:729] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:13:50.360705 16023 consensus_queue.cc:260] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:50.361328 16023 raft_consensus.cc:3037] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:13:50.365546 16023 raft_consensus.cc:2454] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 2.
I20240501 08:13:50.366909 15691 leader_election.cc:302] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:13:50.367241 28927 raft_consensus.cc:2781] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:13:50.369707 13104 raft_consensus.cc:2454] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 982f49fc87f149439cda0de2261b6bfd in term 2.
I20240501 08:13:50.370223 28927 raft_consensus.cc:686] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [term 2 LEADER]: Becoming Leader. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Running, Role: LEADER
I20240501 08:13:50.370661 28927 consensus_queue.cc:237] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:50.377553 12828 catalog_manager.cc:5484] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd reported cstate change: term changed from 1 to 2, leader changed from 5dd722c770d34d6c9154a87917117161 (127.6.56.71) to 982f49fc87f149439cda0de2261b6bfd (127.6.56.69). New cstate: current_term: 2 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:50.576913 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:50.618470 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:50.622349 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:50.679589 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:50.716892 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:50.720568 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:50.765009 16154 raft_consensus.cc:1261] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:50.775697 28927 consensus_queue.cc:1035] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:50.789722 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:50.791025 13104 raft_consensus.cc:1261] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:50.791616 28927 consensus_queue.cc:1035] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:50.795612 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:50.798941 16024 raft_consensus.cc:1261] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 08:13:50.799489 29016 consensus_queue.cc:1035] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240501 08:13:50.900355 15826 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 80612.2us
W20240501 08:13:50.917523 15691 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 55731.4us
W20240501 08:13:50.933678 15689 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 72128.5us
I20240501 08:13:51.239871 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:51.321439 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:51.322413 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:51.347683 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:51.358099 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:51.364727 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:51.368522 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:51.401995 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:51.493521 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(e6fefd76544642ddb787cc6febe8df82): perf score=0.033334
I20240501 08:13:51.528180 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(94e7fb526fbc4ebc94875c00b49ac43d): perf score=0.033343
I20240501 08:13:51.531417 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(cd9a5b5333d54f6d9d8186e4d9b1b1be): perf score=0.033334
I20240501 08:13:51.556749 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(e6fefd76544642ddb787cc6febe8df82) complete. Timing: real 0.063s user 0.035s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":355,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1364,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":813,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":490,"threads_started":2,"wal-append.queue_time_us":322}
I20240501 08:13:51.559794 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(cd9a5b5333d54f6d9d8186e4d9b1b1be): perf score=0.033347
I20240501 08:13:51.576443 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(94e7fb526fbc4ebc94875c00b49ac43d): perf score=0.033345
I20240501 08:13:51.641716 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(94e7fb526fbc4ebc94875c00b49ac43d) complete. Timing: real 0.113s user 0.033s sys 0.023s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6381,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":24287,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1254,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":26495,"threads_started":2,"wal-append.queue_time_us":10659}
I20240501 08:13:51.644615 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(94e7fb526fbc4ebc94875c00b49ac43d): 4834 bytes on disk
I20240501 08:13:51.644470 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(cd9a5b5333d54f6d9d8186e4d9b1b1be) complete. Timing: real 0.112s user 0.048s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":25647,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":4672,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":855,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":49653,"threads_started":2,"wal-append.queue_time_us":12185}
I20240501 08:13:51.648099 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(cd9a5b5333d54f6d9d8186e4d9b1b1be): 4919 bytes on disk
I20240501 08:13:51.648867 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(94e7fb526fbc4ebc94875c00b49ac43d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
I20240501 08:13:51.651538 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(cd9a5b5333d54f6d9d8186e4d9b1b1be) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":717,"lbm_reads_lt_1ms":4}
I20240501 08:13:51.709481 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(94e7fb526fbc4ebc94875c00b49ac43d) complete. Timing: real 0.133s user 0.041s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11708,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":6379,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":919,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":32123,"threads_started":2,"wal-append.queue_time_us":19978}
I20240501 08:13:51.712682 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(94e7fb526fbc4ebc94875c00b49ac43d): 4834 bytes on disk
I20240501 08:13:51.715930 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(94e7fb526fbc4ebc94875c00b49ac43d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4}
I20240501 08:13:51.721552 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(e6fefd76544642ddb787cc6febe8df82): perf score=0.033386
I20240501 08:13:51.737161 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(cd9a5b5333d54f6d9d8186e4d9b1b1be) complete. Timing: real 0.176s user 0.074s sys 0.039s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1274,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1069,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":41052,"threads_started":1,"wal-append.queue_time_us":40066}
I20240501 08:13:51.739899 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(cd9a5b5333d54f6d9d8186e4d9b1b1be): 4919 bytes on disk
I20240501 08:13:51.740515 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(cd9a5b5333d54f6d9d8186e4d9b1b1be) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 08:13:51.742939 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(e6fefd76544642ddb787cc6febe8df82): 4581 bytes on disk
I20240501 08:13:51.743512 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(e6fefd76544642ddb787cc6febe8df82) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 08:13:51.787148 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(e6fefd76544642ddb787cc6febe8df82) complete. Timing: real 0.065s user 0.038s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1305,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2009,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1106,"threads_started":1,"wal-append.queue_time_us":1204}
I20240501 08:13:51.789430 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(e6fefd76544642ddb787cc6febe8df82): 4581 bytes on disk
I20240501 08:13:51.789994 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(e6fefd76544642ddb787cc6febe8df82) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 08:13:51.808993 12971 consensus_queue.cc:237] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:51.811563 13363 raft_consensus.cc:1261] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:51.811821 15755 raft_consensus.cc:1261] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:51.812387 29166 consensus_queue.cc:1035] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:51.812958 29166 consensus_queue.cc:1035] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:51.816201 29166 raft_consensus.cc:2932] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:51.817255 15753 raft_consensus.cc:2932] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:51.831061 12824 catalog_manager.cc:5484] T 714a84eae9324caeb4f25e798d6c35c2 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:51.835033 13363 raft_consensus.cc:2932] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:51.855222 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 714a84eae9324caeb4f25e798d6c35c2 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:13:51.855942 29185 tablet_replica.cc:330] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:13:51.856189 29185 raft_consensus.cc:2229] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:51.856627 29185 raft_consensus.cc:2258] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:51.857321 29185 ts_tablet_manager.cc:1875] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:51.863018 29185 ts_tablet_manager.cc:1888] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:51.863142 29185 log.cc:1196] T 714a84eae9324caeb4f25e798d6c35c2 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/714a84eae9324caeb4f25e798d6c35c2
I20240501 08:13:51.863811 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 714a84eae9324caeb4f25e798d6c35c2 (table rebalancer_extra_table_18 [id=61bd792f42de428aa3ccb21206a71315]) successfully deleted
I20240501 08:13:52.209033 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:52.211970 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:52.219223 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:52.222494 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:52.258615 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:52.296736 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:52.313576 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:52.316992 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:52.478551 13364 consensus_queue.cc:237] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:13:52.480635 13104 raft_consensus.cc:1261] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:52.480629 16154 raft_consensus.cc:1261] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:52.481119 28843 consensus_queue.cc:1035] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:52.481562 28995 consensus_queue.cc:1035] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:52.484165 28995 raft_consensus.cc:2932] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:52.484858 16154 raft_consensus.cc:2932] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:52.487430 13104 raft_consensus.cc:2932] T a328773ef57444cb8147524107b15dba P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:52.491308 12830 catalog_manager.cc:5484] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 4 to 5, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:13:52.507177 12953 tablet_service.cc:1514] Processing DeleteTablet for tablet a328773ef57444cb8147524107b15dba with delete_type TABLET_DATA_TOMBSTONED (TS 76d830e199eb461c8ca689c9460ee0c2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57838
I20240501 08:13:52.509531 29264 tablet_replica.cc:330] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2: stopping tablet replica
I20240501 08:13:52.510030 29264 raft_consensus.cc:2229] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:52.510802 29264 raft_consensus.cc:2258] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:52.511629 29264 ts_tablet_manager.cc:1875] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:52.518281 29264 ts_tablet_manager.cc:1888] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:13:52.518468 29264 log.cc:1196] T a328773ef57444cb8147524107b15dba P 76d830e199eb461c8ca689c9460ee0c2: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/wals/a328773ef57444cb8147524107b15dba
I20240501 08:13:52.519627 12820 catalog_manager.cc:4830] TS 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): tablet a328773ef57444cb8147524107b15dba (table rebalance_test_table_11 [id=16e5940bd1584cea8c5869059eedc2aa]) successfully deleted
I20240501 08:13:52.823959 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:52.830119 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:52.872363 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:52.896852 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:52.943436 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:52.967051 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:53.030390 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:53.073010 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:53.218060 12971 consensus_queue.cc:237] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:53.220436 16023 raft_consensus.cc:1261] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:53.220594 16157 raft_consensus.cc:1261] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:53.221174 29167 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:53.221451 29353 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:53.224630 29353 raft_consensus.cc:2932] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:53.225737 16023 raft_consensus.cc:2932] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:53.225828 16157 raft_consensus.cc:2932] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:53.231348 12829 catalog_manager.cc:5484] T cf908880298b462391a84f5a8d4e3a6b P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 2 to 4, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) evicted. New cstate: current_term: 2 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:53.268545 15736 tablet_service.cc:1514] Processing DeleteTablet for tablet cf908880298b462391a84f5a8d4e3a6b with delete_type TABLET_DATA_TOMBSTONED (TS 982f49fc87f149439cda0de2261b6bfd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40166
I20240501 08:13:53.269747 29365 tablet_replica.cc:330] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: stopping tablet replica
I20240501 08:13:53.270044 29365 raft_consensus.cc:2229] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:53.270284 29365 raft_consensus.cc:2258] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:53.271107 29365 ts_tablet_manager.cc:1875] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:53.276499 29365 ts_tablet_manager.cc:1888] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 08:13:53.276619 29365 log.cc:1196] T cf908880298b462391a84f5a8d4e3a6b P 982f49fc87f149439cda0de2261b6bfd: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/wals/cf908880298b462391a84f5a8d4e3a6b
I20240501 08:13:53.277396 12819 catalog_manager.cc:4830] TS 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): tablet cf908880298b462391a84f5a8d4e3a6b (table rebalancer_extra_table_12 [id=5ec25bc622944a14a3f9a0ce15683b4f]) successfully deleted
I20240501 08:13:53.512400 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:53.529489 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:53.543604 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:53.545406 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:53.582619 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:53.599117 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:53.669965 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:53.711028 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:53.803892 16023 consensus_queue.cc:237] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:53.806054 13363 raft_consensus.cc:1261] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:53.806272 12971 raft_consensus.cc:1261] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:53.806661 28911 consensus_queue.cc:1035] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:53.806912 29412 consensus_queue.cc:1035] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:53.809698 29412 raft_consensus.cc:2932] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:53.810199 13364 raft_consensus.cc:2932] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:53.817569 12831 catalog_manager.cc:5484] T 4345eabfdb894def8d7063773dc5fac6 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:53.827632 12971 raft_consensus.cc:2932] T 4345eabfdb894def8d7063773dc5fac6 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:53.838979 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 4345eabfdb894def8d7063773dc5fac6 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:13:53.839560 29453 tablet_replica.cc:330] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:13:53.840209 29453 raft_consensus.cc:2229] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:53.840464 29453 raft_consensus.cc:2258] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:53.841126 29453 ts_tablet_manager.cc:1875] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:53.846369 29453 ts_tablet_manager.cc:1888] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:53.846572 29453 log.cc:1196] T 4345eabfdb894def8d7063773dc5fac6 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/4345eabfdb894def8d7063773dc5fac6
I20240501 08:13:53.847364 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 4345eabfdb894def8d7063773dc5fac6 (table rebalancer_extra_table_50 [id=7922e537ada34a4a8f32d249d53634e0]) successfully deleted
I20240501 08:13:54.184074 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:54.188915 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:54.208546 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:54.210347 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:54.272516 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:54.304080 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:54.343617 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:54.385550 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:54.486960 13235 consensus_queue.cc:237] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:13:54.488855 16024 raft_consensus.cc:1261] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:54.488916 16154 raft_consensus.cc:1261] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:54.489348 29324 consensus_queue.cc:1035] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:54.489653 28458 consensus_queue.cc:1035] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:54.492383 28458 raft_consensus.cc:2932] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:54.492861 16024 raft_consensus.cc:2932] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:54.493672 16154 raft_consensus.cc:2932] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:54.499563 12833 catalog_manager.cc:5484] T a713c94c6bb14d8793ee429e52f0e7a0 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 2 to 4, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 2 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:13:54.546036 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet a713c94c6bb14d8793ee429e52f0e7a0 with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56180
I20240501 08:13:54.550868 29566 tablet_replica.cc:330] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:13:54.551144 29566 raft_consensus.cc:2229] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:54.551323 29566 raft_consensus.cc:2258] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:54.552016 29566 ts_tablet_manager.cc:1875] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:54.558784 29566 ts_tablet_manager.cc:1888] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 08:13:54.558952 29566 log.cc:1196] T a713c94c6bb14d8793ee429e52f0e7a0 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/a713c94c6bb14d8793ee429e52f0e7a0
I20240501 08:13:54.560046 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet a713c94c6bb14d8793ee429e52f0e7a0 (table rebalancer_extra_table_48 [id=834d37fb225541508b80d79516607e28]) successfully deleted
I20240501 08:13:54.679054 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(ec6d272070b140d9b918d182613008f4): perf score=0.033344
I20240501 08:13:54.689154 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(d1af72684b304cab9a7d960084ea5926): perf score=0.033342
I20240501 08:13:54.778442 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(ec6d272070b140d9b918d182613008f4): perf score=0.033372
I20240501 08:13:54.788633 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(d1af72684b304cab9a7d960084ea5926) complete. Timing: real 0.099s user 0.050s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":14344,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":7622,"drs_written":1,"lbm_read_time_us":175,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":21167,"threads_started":2,"wal-append.queue_time_us":6554}
I20240501 08:13:54.792661 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(d1af72684b304cab9a7d960084ea5926): 4834 bytes on disk
I20240501 08:13:54.794915 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(d1af72684b304cab9a7d960084ea5926) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4}
I20240501 08:13:54.799360 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(ec6d272070b140d9b918d182613008f4) complete. Timing: real 0.120s user 0.032s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":22586,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":5609,"drs_written":1,"lbm_read_time_us":187,"lbm_reads_lt_1ms":4,"lbm_write_time_us":826,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28439,"threads_started":2,"wal-append.queue_time_us":11628}
I20240501 08:13:54.802407 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(ec6d272070b140d9b918d182613008f4): 4581 bytes on disk
I20240501 08:13:54.803169 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(ec6d272070b140d9b918d182613008f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240501 08:13:54.805004 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(53872527ad264dd9b4b1627ff8aa46db): perf score=0.033375
I20240501 08:13:54.831650 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(53872527ad264dd9b4b1627ff8aa46db): perf score=0.033378
I20240501 08:13:54.891160 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(ec6d272070b140d9b918d182613008f4) complete. Timing: real 0.112s user 0.054s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":10344,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":8446,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":3781,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":15847,"threads_started":2,"wal-append.queue_time_us":5238}
I20240501 08:13:54.904237 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(d1af72684b304cab9a7d960084ea5926): perf score=0.033399
I20240501 08:13:54.960208 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:55.053368 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(53872527ad264dd9b4b1627ff8aa46db) complete. Timing: real 0.248s user 0.046s sys 0.051s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":3768,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":983,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":76991,"threads_started":1,"wal-append.queue_time_us":74398}
I20240501 08:13:55.058255 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(53872527ad264dd9b4b1627ff8aa46db): 4921 bytes on disk
I20240501 08:13:55.060425 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(53872527ad264dd9b4b1627ff8aa46db) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240501 08:13:55.083195 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:55.083747 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:55.132625 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:55.171013 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:55.218578 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:55.225102 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
W20240501 08:13:55.236409 13172 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.322s user 0.025s sys 0.076s
I20240501 08:13:55.244031 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(53872527ad264dd9b4b1627ff8aa46db) complete. Timing: real 0.412s user 0.053s sys 0.100s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14584,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":6161,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":892,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":336206,"threads_started":2,"wal-append.queue_time_us":312242}
I20240501 08:13:55.247221 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(53872527ad264dd9b4b1627ff8aa46db): 4921 bytes on disk
I20240501 08:13:55.248008 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(53872527ad264dd9b4b1627ff8aa46db) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240501 08:13:55.250977 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:13:55.252688 13302 tablet_replica.cc:1402] Time spent flushing the Log queue took a long time: real 0.214s user 0.024s sys 0.042s
I20240501 08:13:55.256603 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(d1af72684b304cab9a7d960084ea5926) complete. Timing: real 0.340s user 0.064s sys 0.050s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":1490,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1577,"drs_written":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4,"lbm_write_time_us":914,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":213608,"threads_started":1,"wal-append.queue_time_us":213712}
I20240501 08:13:55.261607 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(d1af72684b304cab9a7d960084ea5926): 4834 bytes on disk
I20240501 08:13:55.262245 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(d1af72684b304cab9a7d960084ea5926) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4}
I20240501 08:13:55.264765 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(ec6d272070b140d9b918d182613008f4): 4581 bytes on disk
I20240501 08:13:55.273236 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(ec6d272070b140d9b918d182613008f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240501 08:13:55.422502 15755 consensus_queue.cc:237] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:13:55.424396 16154 raft_consensus.cc:1261] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:55.424607 13103 raft_consensus.cc:1261] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:13:55.425328 29070 consensus_queue.cc:1035] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:55.425613 29335 consensus_queue.cc:1035] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:55.428336 29652 raft_consensus.cc:2932] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:55.428862 16154 raft_consensus.cc:2932] T b320e884c36d461ba5f2941d9d76185f P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:55.428961 13103 raft_consensus.cc:2932] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:55.435397 12825 catalog_manager.cc:5484] T b320e884c36d461ba5f2941d9d76185f P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 2 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) evicted. New cstate: current_term: 2 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:13:55.442114 16005 tablet_service.cc:1514] Processing DeleteTablet for tablet b320e884c36d461ba5f2941d9d76185f with delete_type TABLET_DATA_TOMBSTONED (TS 5dd722c770d34d6c9154a87917117161 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50326
I20240501 08:13:55.449498 29656 tablet_replica.cc:330] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: stopping tablet replica
I20240501 08:13:55.449728 29656 raft_consensus.cc:2229] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:55.449910 29656 raft_consensus.cc:2258] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:55.450469 29656 ts_tablet_manager.cc:1875] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:55.455976 29656 ts_tablet_manager.cc:1888] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240501 08:13:55.456075 29656 log.cc:1196] T b320e884c36d461ba5f2941d9d76185f P 5dd722c770d34d6c9154a87917117161: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/wals/b320e884c36d461ba5f2941d9d76185f
I20240501 08:13:55.456880 12819 catalog_manager.cc:4830] TS 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): tablet b320e884c36d461ba5f2941d9d76185f (table rebalancer_extra_table_26 [id=c9f710e4dc7e4a74a731149a0d32483e]) successfully deleted
I20240501 08:13:55.536799 13363 consensus_queue.cc:237] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:55.540095 16154 raft_consensus.cc:1261] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:55.540751 15892 raft_consensus.cc:1261] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:55.540678 29688 consensus_queue.cc:1035] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:55.541855 13301 consensus_peers.cc:485] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7da2efbf509e4d8c82acc7b12fc6f803. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:55.542690 28843 consensus_queue.cc:1035] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:55.581559 28843 raft_consensus.cc:2932] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.582176 15892 raft_consensus.cc:2932] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.589509 12825 catalog_manager.cc:5484] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.593122 16157 raft_consensus.cc:2932] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.662101 15755 consensus_queue.cc:237] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:55.664259 13232 raft_consensus.cc:1261] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:55.664419 15691 consensus_peers.cc:485] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 608bc35666234d2cbcb816439d050196. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:55.664618 15892 raft_consensus.cc:1261] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:55.665089 29335 consensus_queue.cc:1035] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:55.665347 29070 consensus_queue.cc:1035] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:55.668538 29335 raft_consensus.cc:2932] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:55.669167 13232 raft_consensus.cc:2932] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:55.672230 15892 raft_consensus.cc:2932] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:55.677739 12831 catalog_manager.cc:5484] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:55.709939 13103 consensus_queue.cc:237] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:55.712595 12973 raft_consensus.cc:1261] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:13:55.712632 13363 raft_consensus.cc:1261] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:13:55.713047 13040 consensus_peers.cc:485] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db4717fa6c4e45d18702741c141f6d12. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:55.713315 29727 consensus_queue.cc:1035] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:55.713688 29524 consensus_queue.cc:1035] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:13:55.716965 29524 raft_consensus.cc:2932] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:55.717487 12973 raft_consensus.cc:2932] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:55.721400 13363 raft_consensus.cc:2932] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:55.725546 12832 catalog_manager.cc:5484] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:55.782202 13363 consensus_queue.cc:237] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:55.784543 16157 raft_consensus.cc:1261] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:55.785272 28843 consensus_queue.cc:1035] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:13:55.785638 13301 consensus_peers.cc:485] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f795db6dcb644d691c3ba9414a62c51. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:55.786159 15755 raft_consensus.cc:1261] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:55.787505 29530 consensus_queue.cc:1035] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:55.789994 29530 raft_consensus.cc:2932] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.790565 15755 raft_consensus.cc:2932] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.796635 16157 raft_consensus.cc:2932] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.798777 12825 catalog_manager.cc:5484] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.834599 13103 consensus_queue.cc:237] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:55.836967 13232 raft_consensus.cc:1261] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:55.836956 15893 raft_consensus.cc:1261] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:55.837072 13040 consensus_peers.cc:485] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b5fda95d9e9d48e7a4d6eb4d9b85fb57. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:55.837461 29524 consensus_queue.cc:1035] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:55.837747 29730 consensus_queue.cc:1035] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:55.841176 29730 raft_consensus.cc:2932] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:55.841792 13232 raft_consensus.cc:2932] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:55.845173 15893 raft_consensus.cc:2932] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:55.848098 12832 catalog_manager.cc:5484] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 2, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:55.893522 15755 consensus_queue.cc:237] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:55.896579 16023 raft_consensus.cc:1261] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:55.896554 16157 raft_consensus.cc:1261] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:55.897045 15692 consensus_peers.cc:485] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cca8e4588be4791800840a638bf0372. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:55.897171 29335 consensus_queue.cc:1035] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:55.897424 29652 consensus_queue.cc:1035] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:55.901521 29652 raft_consensus.cc:2932] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.908126 16023 raft_consensus.cc:2932] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.914506 16157 raft_consensus.cc:2932] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:55.917955 12832 catalog_manager.cc:5484] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:56.011065 15755 consensus_queue.cc:237] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:56.013728 16157 raft_consensus.cc:1261] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:13:56.013839 15692 consensus_peers.cc:485] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3549f34428b34ea3920f0f0f20c24524. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:13:56.014258 15893 raft_consensus.cc:1261] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:13:56.014341 29335 consensus_queue.cc:1035] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:56.014909 29335 consensus_queue.cc:1035] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:13:56.018146 29652 raft_consensus.cc:2932] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:56.021288 15893 raft_consensus.cc:2932] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:56.023058 16157 raft_consensus.cc:2932] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:56.027441 12832 catalog_manager.cc:5484] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:56.064405 29806 ts_tablet_manager.cc:902] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
W20240501 08:13:56.090011 13168 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 55414.2us
I20240501 08:13:56.094218 29810 ts_tablet_manager.cc:902] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:56.100821 29806 tablet_copy_client.cc:308] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:13:56.102001 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet 608bc35666234d2cbcb816439d050196 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:35425)
I20240501 08:13:56.102151 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet 608bc35666234d2cbcb816439d050196 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:35425: session id = 4ecb724cb4764a53988fde00407db4c0-608bc35666234d2cbcb816439d050196
I20240501 08:13:56.107576 15776 tablet_copy_source_session.cc:205] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:56.109272 29806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 608bc35666234d2cbcb816439d050196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:56.110927 29810 tablet_copy_client.cc:308] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:56.111567 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 7da2efbf509e4d8c82acc7b12fc6f803 from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:39327)
I20240501 08:13:56.111716 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 7da2efbf509e4d8c82acc7b12fc6f803 from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:39327: session id = 76d830e199eb461c8ca689c9460ee0c2-7da2efbf509e4d8c82acc7b12fc6f803
I20240501 08:13:56.114324 13385 tablet_copy_source_session.cc:205] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:56.117018 29810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da2efbf509e4d8c82acc7b12fc6f803. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:56.117873 29806 tablet_copy_client.cc:783] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:56.118150 29806 tablet_copy_client.cc:647] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:56.119827 29806 tablet_copy_client.cc:523] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:56.125605 29810 tablet_copy_client.cc:783] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:56.125725 29806 tablet_bootstrap.cc:492] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:56.125805 29810 tablet_copy_client.cc:647] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:56.127537 29810 tablet_copy_client.cc:523] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:56.130460 29810 tablet_bootstrap.cc:492] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:56.137125 29806 tablet_bootstrap.cc:492] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:56.137626 29806 tablet_bootstrap.cc:492] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:56.137964 29806 ts_tablet_manager.cc:1367] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240501 08:13:56.138028 29810 tablet_bootstrap.cc:492] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:56.138351 29810 tablet_bootstrap.cc:492] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:56.138556 29810 ts_tablet_manager.cc:1367] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:13:56.138724 29806 raft_consensus.cc:348] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:56.138959 29806 raft_consensus.cc:729] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:56.139133 29810 raft_consensus.cc:348] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:56.139315 29810 raft_consensus.cc:729] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:56.139248 29806 consensus_queue.cc:260] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:56.139621 29810 consensus_queue.cc:260] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:56.140246 29810 ts_tablet_manager.cc:1398] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:13:56.141049 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-7da2efbf509e4d8c82acc7b12fc6f803 received from {username='slave'} at 127.6.56.65:39327
I20240501 08:13:56.141166 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-7da2efbf509e4d8c82acc7b12fc6f803 on tablet 7da2efbf509e4d8c82acc7b12fc6f803 with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:13:56.141351 29806 ts_tablet_manager.cc:1398] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:13:56.151055 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-608bc35666234d2cbcb816439d050196 received from {username='slave'} at 127.6.56.68:35425
I20240501 08:13:56.151188 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-608bc35666234d2cbcb816439d050196 on tablet 608bc35666234d2cbcb816439d050196 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:56.292971 29810 ts_tablet_manager.cc:902] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:13:56.297343 29855 ts_tablet_manager.cc:902] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:13:56.316905 29810 tablet_copy_client.cc:308] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.68:36991
I20240501 08:13:56.319128 13385 tablet_copy_service.cc:140] P 4ecb724cb4764a53988fde00407db4c0: Received BeginTabletCopySession request for tablet 2f795db6dcb644d691c3ba9414a62c51 from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:39327)
I20240501 08:13:56.319370 13385 tablet_copy_service.cc:161] P 4ecb724cb4764a53988fde00407db4c0: Beginning new tablet copy session on tablet 2f795db6dcb644d691c3ba9414a62c51 from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:39327: session id = 76d830e199eb461c8ca689c9460ee0c2-2f795db6dcb644d691c3ba9414a62c51
I20240501 08:13:56.322314 13385 tablet_copy_source_session.cc:205] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:56.323843 29810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f795db6dcb644d691c3ba9414a62c51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:56.332039 29810 tablet_copy_client.cc:783] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:56.332429 29810 tablet_copy_client.cc:647] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:56.332975 29806 ts_tablet_manager.cc:902] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:13:56.334919 29810 tablet_copy_client.cc:523] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:56.340000 29810 tablet_bootstrap.cc:492] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:56.372648 29810 tablet_bootstrap.cc:492] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:56.373940 29810 tablet_bootstrap.cc:492] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:56.374981 29810 ts_tablet_manager.cc:1367] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.001s
I20240501 08:13:56.376482 29810 raft_consensus.cc:348] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:56.379156 29810 raft_consensus.cc:729] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:56.379956 29810 consensus_queue.cc:260] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:56.381063 29806 tablet_copy_client.cc:308] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:13:56.381822 13124 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57 from peer 4ecb724cb4764a53988fde00407db4c0 ({username='slave'} at 127.6.56.68:60159)
I20240501 08:13:56.381990 13124 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57 from peer 4ecb724cb4764a53988fde00407db4c0 at {username='slave'} at 127.6.56.68:60159: session id = 4ecb724cb4764a53988fde00407db4c0-b5fda95d9e9d48e7a4d6eb4d9b85fb57
I20240501 08:13:56.384783 13124 tablet_copy_source_session.cc:205] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:56.390350 29806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:56.394569 29855 tablet_copy_client.cc:308] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:13:56.394289 29810 ts_tablet_manager.cc:1398] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240501 08:13:56.395255 13124 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet db4717fa6c4e45d18702741c141f6d12 from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:42231)
I20240501 08:13:56.395412 13124 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet db4717fa6c4e45d18702741c141f6d12 from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:42231: session id = 5dd722c770d34d6c9154a87917117161-db4717fa6c4e45d18702741c141f6d12
I20240501 08:13:56.395622 13385 tablet_copy_service.cc:342] P 4ecb724cb4764a53988fde00407db4c0: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-2f795db6dcb644d691c3ba9414a62c51 received from {username='slave'} at 127.6.56.65:39327
I20240501 08:13:56.395742 13385 tablet_copy_service.cc:433] P 4ecb724cb4764a53988fde00407db4c0: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-2f795db6dcb644d691c3ba9414a62c51 on tablet 2f795db6dcb644d691c3ba9414a62c51 with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:13:56.397379 29810 ts_tablet_manager.cc:902] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:13:56.398650 13124 tablet_copy_source_session.cc:205] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:56.404989 29855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4717fa6c4e45d18702741c141f6d12. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:56.409957 29806 tablet_copy_client.cc:783] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:56.410723 29806 tablet_copy_client.cc:647] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:56.411717 29855 tablet_copy_client.cc:783] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:56.411926 29855 tablet_copy_client.cc:647] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:56.416420 29876 ts_tablet_manager.cc:902] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:13:56.417817 29806 tablet_copy_client.cc:523] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:56.418465 29855 tablet_copy_client.cc:523] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:56.424372 29855 tablet_bootstrap.cc:492] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:13:56.431972 29810 tablet_copy_client.cc:308] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:13:56.432931 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet 2cca8e4588be4791800840a638bf0372 from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:53631)
I20240501 08:13:56.433115 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet 2cca8e4588be4791800840a638bf0372 from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:53631: session id = 76d830e199eb461c8ca689c9460ee0c2-2cca8e4588be4791800840a638bf0372
I20240501 08:13:56.434370 29806 tablet_bootstrap.cc:492] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap starting.
I20240501 08:13:56.437245 15776 tablet_copy_source_session.cc:205] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:56.439113 29810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cca8e4588be4791800840a638bf0372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:56.451670 29806 tablet_bootstrap.cc:492] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:56.452097 29806 tablet_bootstrap.cc:492] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: Bootstrap complete.
I20240501 08:13:56.452462 29806 ts_tablet_manager.cc:1367] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.007s
I20240501 08:13:56.453639 29806 raft_consensus.cc:348] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:56.453939 29806 raft_consensus.cc:729] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Initialized, Role: FOLLOWER
I20240501 08:13:56.454265 29806 consensus_queue.cc:260] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:56.470866 29876 tablet_copy_client.cc:308] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.69:42897
I20240501 08:13:56.471793 15776 tablet_copy_service.cc:140] P 982f49fc87f149439cda0de2261b6bfd: Received BeginTabletCopySession request for tablet 3549f34428b34ea3920f0f0f20c24524 from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:53631)
I20240501 08:13:56.472129 15776 tablet_copy_service.cc:161] P 982f49fc87f149439cda0de2261b6bfd: Beginning new tablet copy session on tablet 3549f34428b34ea3920f0f0f20c24524 from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:53631: session id = 76d830e199eb461c8ca689c9460ee0c2-3549f34428b34ea3920f0f0f20c24524
I20240501 08:13:56.476379 15776 tablet_copy_source_session.cc:205] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:13:56.479905 29876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3549f34428b34ea3920f0f0f20c24524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:13:56.481986 29806 ts_tablet_manager.cc:1398] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20240501 08:13:56.484757 13124 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 4ecb724cb4764a53988fde00407db4c0-b5fda95d9e9d48e7a4d6eb4d9b85fb57 received from {username='slave'} at 127.6.56.68:60159
I20240501 08:13:56.484898 13124 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 4ecb724cb4764a53988fde00407db4c0-b5fda95d9e9d48e7a4d6eb4d9b85fb57 on tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57 with peer 4ecb724cb4764a53988fde00407db4c0
I20240501 08:13:56.485363 29810 tablet_copy_client.cc:783] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:56.492568 29810 tablet_copy_client.cc:647] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:56.494484 29810 tablet_copy_client.cc:523] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:56.524369 29810 tablet_bootstrap.cc:492] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:56.548780 29876 tablet_copy_client.cc:783] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:13:56.549326 29876 tablet_copy_client.cc:647] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:13:56.554037 29876 tablet_copy_client.cc:523] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:13:56.563809 29876 tablet_bootstrap.cc:492] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:13:56.574349 29810 tablet_bootstrap.cc:492] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:56.579805 29810 tablet_bootstrap.cc:492] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:56.580597 29810 ts_tablet_manager.cc:1367] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.000s
I20240501 08:13:56.590627 29810 raft_consensus.cc:348] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:56.591017 29810 raft_consensus.cc:729] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:56.596465 29810 consensus_queue.cc:260] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:56.599647 13363 raft_consensus.cc:1203] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:56.599949 29810 ts_tablet_manager.cc:1398] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240501 08:13:56.600661 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-2cca8e4588be4791800840a638bf0372 received from {username='slave'} at 127.6.56.65:53631
I20240501 08:13:56.600790 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-2cca8e4588be4791800840a638bf0372 on tablet 2cca8e4588be4791800840a638bf0372 with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:13:56.613404 12973 raft_consensus.cc:1203] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:56.616197 29876 tablet_bootstrap.cc:492] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:56.619901 29876 tablet_bootstrap.cc:492] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:13:56.620676 29876 ts_tablet_manager.cc:1367] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.000s
I20240501 08:13:56.622884 29876 raft_consensus.cc:348] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:56.623690 29876 raft_consensus.cc:729] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:13:56.624465 29876 consensus_queue.cc:260] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:56.627481 29876 ts_tablet_manager.cc:1398] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:13:56.628315 15776 tablet_copy_service.cc:342] P 982f49fc87f149439cda0de2261b6bfd: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-3549f34428b34ea3920f0f0f20c24524 received from {username='slave'} at 127.6.56.65:53631
I20240501 08:13:56.628458 15776 tablet_copy_service.cc:433] P 982f49fc87f149439cda0de2261b6bfd: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-3549f34428b34ea3920f0f0f20c24524 on tablet 3549f34428b34ea3920f0f0f20c24524 with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:13:56.674206 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:56.683584 29855 tablet_bootstrap.cc:492] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:13:56.687413 29855 tablet_bootstrap.cc:492] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:13:56.687813 29855 ts_tablet_manager.cc:1367] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.264s user 0.058s sys 0.000s
I20240501 08:13:56.688678 29855 raft_consensus.cc:348] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:56.690356 29855 raft_consensus.cc:729] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:13:56.690654 29855 consensus_queue.cc:260] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:56.691396 29855 ts_tablet_manager.cc:1398] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:13:56.701273 13124 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-db4717fa6c4e45d18702741c141f6d12 received from {username='slave'} at 127.6.56.71:42231
I20240501 08:13:56.701393 13124 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 5dd722c770d34d6c9154a87917117161-db4717fa6c4e45d18702741c141f6d12 on tablet db4717fa6c4e45d18702741c141f6d12 with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:13:56.711387 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:56.723079 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:56.724804 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:56.731729 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:56.744674 16023 raft_consensus.cc:1203] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:13:56.761240 13364 raft_consensus.cc:1203] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:56.797289 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:56.838996 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:13:56.843398 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.71:36997, user_credentials={real_user=slave}} blocked reactor thread for 69302.8us
I20240501 08:13:56.863083 12973 raft_consensus.cc:1203] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:56.876616 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:56.898597 12973 raft_consensus.cc:1203] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:56.983691 12973 raft_consensus.cc:1203] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:13:57.018800 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(427bc3b61a874f83bc27d9be24732e00): perf score=0.033348
I20240501 08:13:57.045229 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(9348009ae1444e73b6120161309dc4a2): perf score=0.033349
I20240501 08:13:57.069155 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(427bc3b61a874f83bc27d9be24732e00): perf score=0.033357
I20240501 08:13:57.088475 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(a412b8ae0aa648d7979fac58b50d86a6): perf score=0.033367
I20240501 08:13:57.129998 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(427bc3b61a874f83bc27d9be24732e00) complete. Timing: real 0.111s user 0.071s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6447,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":16761,"drs_written":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4,"lbm_write_time_us":807,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":27513,"threads_started":2,"wal-append.queue_time_us":21087}
I20240501 08:13:57.132844 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(9348009ae1444e73b6120161309dc4a2): perf score=0.033371
I20240501 08:13:57.144860 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(427bc3b61a874f83bc27d9be24732e00) complete. Timing: real 0.075s user 0.038s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8423,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":6409,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":754,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17134,"threads_started":2,"wal-append.queue_time_us":8436}
I20240501 08:13:57.147572 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(427bc3b61a874f83bc27d9be24732e00): 4579 bytes on disk
I20240501 08:13:57.148423 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(427bc3b61a874f83bc27d9be24732e00) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 08:13:57.182595 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(9348009ae1444e73b6120161309dc4a2) complete. Timing: real 0.137s user 0.033s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":986,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":4230,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5260,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12787,"threads_started":2,"wal-append.queue_time_us":11126}
I20240501 08:13:57.185943 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(a412b8ae0aa648d7979fac58b50d86a6): perf score=0.033386
I20240501 08:13:57.206622 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(9348009ae1444e73b6120161309dc4a2) complete. Timing: real 0.073s user 0.046s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":65,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":3622,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":912,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14371,"threads_started":1,"wal-append.queue_time_us":14460}
I20240501 08:13:57.209139 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(9348009ae1444e73b6120161309dc4a2): 4921 bytes on disk
I20240501 08:13:57.209748 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(9348009ae1444e73b6120161309dc4a2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
I20240501 08:13:57.211666 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(427bc3b61a874f83bc27d9be24732e00): 4579 bytes on disk
I20240501 08:13:57.212183 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(427bc3b61a874f83bc27d9be24732e00) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 08:13:57.213541 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(a412b8ae0aa648d7979fac58b50d86a6) complete. Timing: real 0.125s user 0.049s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7107,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":11183,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1214,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":12748,"threads_started":2,"wal-append.queue_time_us":5440}
I20240501 08:13:57.216006 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(a412b8ae0aa648d7979fac58b50d86a6): 4834 bytes on disk
I20240501 08:13:57.216567 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(a412b8ae0aa648d7979fac58b50d86a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
I20240501 08:13:57.244830 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(a412b8ae0aa648d7979fac58b50d86a6) complete. Timing: real 0.057s user 0.039s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":276,"dirs.run_wall_time_us":1377,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":229,"threads_started":1,"wal-append.queue_time_us":308}
I20240501 08:13:57.249351 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(9348009ae1444e73b6120161309dc4a2): 4921 bytes on disk
I20240501 08:13:57.252928 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(9348009ae1444e73b6120161309dc4a2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 08:13:57.257081 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(a412b8ae0aa648d7979fac58b50d86a6): 4834 bytes on disk
I20240501 08:13:57.257769 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(a412b8ae0aa648d7979fac58b50d86a6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4}
I20240501 08:13:57.489794 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:57.513702 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:57.582376 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 08:13:57.582969 15691 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 96333.5us
W20240501 08:13:57.592278 15826 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 66389.2us
I20240501 08:13:57.618656 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:57.641042 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:57.686332 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:57.742887 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:57.764762 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:57.893363 15753 consensus_queue.cc:237] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:57.895560 16024 raft_consensus.cc:1261] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:57.895582 12971 raft_consensus.cc:1261] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:57.896131 29652 consensus_queue.cc:1035] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:57.896449 29993 consensus_queue.cc:1035] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:57.902556 29993 raft_consensus.cc:2932] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:57.904996 16024 raft_consensus.cc:2932] T 2cca8e4588be4791800840a638bf0372 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:57.917183 12827 catalog_manager.cc:5484] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:57.920709 12973 raft_consensus.cc:2932] T 2cca8e4588be4791800840a638bf0372 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:57.943295 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cca8e4588be4791800840a638bf0372 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:13:57.949707 30032 tablet_replica.cc:330] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:13:57.951598 30032 raft_consensus.cc:2229] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:57.951982 30032 raft_consensus.cc:2258] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:57.953001 30032 ts_tablet_manager.cc:1875] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:57.966228 30032 ts_tablet_manager.cc:1888] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:57.966485 30032 log.cc:1196] T 2cca8e4588be4791800840a638bf0372 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/2cca8e4588be4791800840a638bf0372
I20240501 08:13:57.967439 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 2cca8e4588be4791800840a638bf0372 (table rebalancer_extra_table_40 [id=303c9e265ecc46bb9362d00ad3589eb3]) successfully deleted
I20240501 08:13:58.251699 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:58.253602 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:58.286384 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:58.313098 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:58.323537 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:58.367054 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:58.396209 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:13:58.408529 13170 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.66:35509, user_credentials={real_user=slave}} blocked reactor thread for 84960.2us
I20240501 08:13:58.423707 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:58.619066 13104 consensus_queue.cc:237] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:13:58.621186 13363 raft_consensus.cc:1261] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:58.621263 15892 raft_consensus.cc:1261] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:58.621752 29727 consensus_queue.cc:1035] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:58.623214 29727 consensus_queue.cc:1035] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:58.628723 29727 raft_consensus.cc:2932] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:58.634521 13364 raft_consensus.cc:2932] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:58.642573 12827 catalog_manager.cc:5484] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:58.645872 15893 raft_consensus.cc:2932] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:13:58.651365 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57 with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34524
I20240501 08:13:58.651990 30116 tablet_replica.cc:330] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:13:58.652195 30116 raft_consensus.cc:2229] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:58.652442 30116 raft_consensus.cc:2258] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:58.653168 30116 ts_tablet_manager.cc:1875] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:58.660792 30116 ts_tablet_manager.cc:1888] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:58.660997 30116 log.cc:1196] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/b5fda95d9e9d48e7a4d6eb4d9b85fb57
I20240501 08:13:58.661775 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57 (table rebalancer_extra_table_16 [id=d149afe167ad401ab5a28770ee66cca1]) successfully deleted
I20240501 08:13:58.897213 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:58.912989 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:58.925892 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:58.952996 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:13:58.979733 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:59.021514 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:59.078641 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:59.124491 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:59.269634 13363 consensus_queue.cc:237] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:13:59.271608 15754 raft_consensus.cc:1261] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:59.271674 12971 raft_consensus.cc:1261] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:13:59.272219 29937 consensus_queue.cc:1035] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:59.272739 30176 consensus_queue.cc:1035] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:13:59.275234 29920 raft_consensus.cc:2932] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:59.275890 15754 raft_consensus.cc:2932] T 2f795db6dcb644d691c3ba9414a62c51 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:59.275962 12971 raft_consensus.cc:2932] T 2f795db6dcb644d691c3ba9414a62c51 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:59.281836 12830 catalog_manager.cc:5484] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:13:59.297108 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f795db6dcb644d691c3ba9414a62c51 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:13:59.300791 30207 tablet_replica.cc:330] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:13:59.301012 30207 raft_consensus.cc:2229] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:59.301244 30207 raft_consensus.cc:2258] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:59.301810 30207 ts_tablet_manager.cc:1875] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:59.307451 30207 ts_tablet_manager.cc:1888] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:13:59.307674 30207 log.cc:1196] T 2f795db6dcb644d691c3ba9414a62c51 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/2f795db6dcb644d691c3ba9414a62c51
I20240501 08:13:59.308483 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 2f795db6dcb644d691c3ba9414a62c51 (table rebalancer_extra_table_24 [id=74cfb51623cd404f91ebe2e0450dcd40]) successfully deleted
I20240501 08:13:59.525756 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:13:59.529049 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:13:59.565837 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:13:59.596530 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:13:59.618345 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:59.619167 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
W20240501 08:13:59.652742 13168 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.72:45383, user_credentials={real_user=slave}} blocked reactor thread for 67671.4us
I20240501 08:13:59.704624 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:13:59.705507 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:13:59.827948 13104 consensus_queue.cc:237] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:13:59.830060 16024 raft_consensus.cc:1261] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:59.830186 12971 raft_consensus.cc:1261] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:13:59.830608 30027 consensus_queue.cc:1035] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:59.833689 30112 consensus_queue.cc:1035] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:13:59.836728 30027 raft_consensus.cc:2932] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:59.838282 12971 raft_consensus.cc:2932] T db4717fa6c4e45d18702741c141f6d12 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:59.838455 16024 raft_consensus.cc:2932] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:59.843677 12831 catalog_manager.cc:5484] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:13:59.866887 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet db4717fa6c4e45d18702741c141f6d12 with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:13:59.868403 30290 tablet_replica.cc:330] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:13:59.868733 30290 raft_consensus.cc:2229] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:13:59.868979 30290 raft_consensus.cc:2258] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:13:59.869752 30290 ts_tablet_manager.cc:1875] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:13:59.876557 30290 ts_tablet_manager.cc:1888] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:13:59.876763 30290 log.cc:1196] T db4717fa6c4e45d18702741c141f6d12 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/db4717fa6c4e45d18702741c141f6d12
I20240501 08:13:59.877516 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet db4717fa6c4e45d18702741c141f6d12 (table rebalance_test_table_13 [id=0980a0d51532489eafb86f16ede323ef]) successfully deleted
I20240501 08:14:00.177791 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(7c4b83a76d7b44039ebb17245fcd8a3c): perf score=0.033346
I20240501 08:14:00.197391 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:00.225350 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:00.250526 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(6fb95e3405ae45d3bfe71fb9d552e3fe): perf score=0.033368
I20240501 08:14:00.253782 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(a328773ef57444cb8147524107b15dba): perf score=0.033366
I20240501 08:14:00.294466 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:00.307732 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(a328773ef57444cb8147524107b15dba): perf score=0.033377
I20240501 08:14:00.366487 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:00.371994 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(7c4b83a76d7b44039ebb17245fcd8a3c) complete. Timing: real 0.194s user 0.046s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7686,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1378,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1158,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":65059,"threads_started":2,"wal-append.queue_time_us":43691}
I20240501 08:14:00.375378 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(7c4b83a76d7b44039ebb17245fcd8a3c): 4832 bytes on disk
I20240501 08:14:00.375474 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:00.380578 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(7c4b83a76d7b44039ebb17245fcd8a3c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240501 08:14:00.386369 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:00.443296 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:00.491197 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:00.624266 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(a328773ef57444cb8147524107b15dba) complete. Timing: real 0.295s user 0.070s sys 0.045s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":99641,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":14340,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1243,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":156870,"threads_started":2,"wal-append.queue_time_us":57196}
I20240501 08:14:00.626735 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(a328773ef57444cb8147524107b15dba): 4921 bytes on disk
I20240501 08:14:00.627403 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(a328773ef57444cb8147524107b15dba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 08:14:00.635531 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(a328773ef57444cb8147524107b15dba) complete. Timing: real 0.381s user 0.068s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":64705,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":20358,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":910,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":117238,"threads_started":2,"wal-append.queue_time_us":52901}
I20240501 08:14:00.638164 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(a328773ef57444cb8147524107b15dba): 4921 bytes on disk
I20240501 08:14:00.639892 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(a328773ef57444cb8147524107b15dba) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":1094,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3}
I20240501 08:14:00.642518 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(6fb95e3405ae45d3bfe71fb9d552e3fe): perf score=0.033470
I20240501 08:14:00.707207 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(6fb95e3405ae45d3bfe71fb9d552e3fe) complete. Timing: real 0.456s user 0.074s sys 0.010s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":18638,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":2514,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5269,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":42273,"threads_started":2,"wal-append.queue_time_us":26953}
I20240501 08:14:00.715168 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(7c4b83a76d7b44039ebb17245fcd8a3c): perf score=0.033491
I20240501 08:14:00.744097 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(6fb95e3405ae45d3bfe71fb9d552e3fe) complete. Timing: real 0.095s user 0.037s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1406,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1051,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":219,"threads_started":1,"wal-append.queue_time_us":315}
I20240501 08:14:00.747289 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(6fb95e3405ae45d3bfe71fb9d552e3fe): 4581 bytes on disk
I20240501 08:14:00.748026 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(6fb95e3405ae45d3bfe71fb9d552e3fe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
I20240501 08:14:00.772981 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(7c4b83a76d7b44039ebb17245fcd8a3c) complete. Timing: real 0.056s user 0.039s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":209,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1349,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":947,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":274,"threads_started":1,"wal-append.queue_time_us":389}
I20240501 08:14:00.775589 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(7c4b83a76d7b44039ebb17245fcd8a3c): 4832 bytes on disk
I20240501 08:14:00.776110 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(7c4b83a76d7b44039ebb17245fcd8a3c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 08:14:00.778044 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(6fb95e3405ae45d3bfe71fb9d552e3fe): 4581 bytes on disk
I20240501 08:14:00.778522 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(6fb95e3405ae45d3bfe71fb9d552e3fe) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240501 08:14:00.787096 15754 consensus_queue.cc:237] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:00.789183 15893 raft_consensus.cc:1261] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:00.789199 13365 raft_consensus.cc:1261] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:00.789758 29993 consensus_queue.cc:1035] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:00.790259 29993 consensus_queue.cc:1035] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:00.792261 30180 raft_consensus.cc:2932] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:14:00.792912 15893 raft_consensus.cc:2932] T 608bc35666234d2cbcb816439d050196 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:14:00.793555 13365 raft_consensus.cc:2932] T 608bc35666234d2cbcb816439d050196 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:14:00.798000 12830 catalog_manager.cc:5484] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index 2 to 3, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) evicted. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:14:00.804678 13213 tablet_service.cc:1514] Processing DeleteTablet for tablet 608bc35666234d2cbcb816439d050196 with delete_type TABLET_DATA_TOMBSTONED (TS 6ed448d04e4646658c0de37d829ff15b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:34524
I20240501 08:14:00.807550 30388 tablet_replica.cc:330] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b: stopping tablet replica
I20240501 08:14:00.807763 30388 raft_consensus.cc:2229] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:00.807988 30388 raft_consensus.cc:2258] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:00.808722 30388 ts_tablet_manager.cc:1875] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:00.822335 30388 ts_tablet_manager.cc:1888] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:14:00.822504 30388 log.cc:1196] T 608bc35666234d2cbcb816439d050196 P 6ed448d04e4646658c0de37d829ff15b: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/wals/608bc35666234d2cbcb816439d050196
I20240501 08:14:00.823479 12820 catalog_manager.cc:4830] TS 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): tablet 608bc35666234d2cbcb816439d050196 (table rebalancer_extra_table_22 [id=48397a21eba44b8592fc40090375f0dd]) successfully deleted
I20240501 08:14:01.219159 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:01.237470 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:01.237920 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:01.265180 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:01.293238 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:01.293475 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:01.356674 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:01.376091 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:14:01.457073 13041 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 86903.2us
I20240501 08:14:01.544441 13365 consensus_queue.cc:237] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:01.549503 12973 raft_consensus.cc:1261] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:01.550094 15893 raft_consensus.cc:1261] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:01.550359 29920 consensus_queue.cc:1035] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:01.552188 29920 consensus_queue.cc:1035] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:01.555347 29920 raft_consensus.cc:2932] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:01.563570 15893 raft_consensus.cc:2932] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:01.564919 12831 catalog_manager.cc:5484] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:01.585314 12973 raft_consensus.cc:2932] T 7da2efbf509e4d8c82acc7b12fc6f803 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:01.610671 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da2efbf509e4d8c82acc7b12fc6f803 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:14:01.626089 30482 tablet_replica.cc:330] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:14:01.626338 30482 raft_consensus.cc:2229] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:01.626550 30482 raft_consensus.cc:2258] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:01.627203 30482 ts_tablet_manager.cc:1875] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:01.635504 30482 ts_tablet_manager.cc:1888] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:14:01.635651 30482 log.cc:1196] T 7da2efbf509e4d8c82acc7b12fc6f803 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/7da2efbf509e4d8c82acc7b12fc6f803
I20240501 08:14:01.636499 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 7da2efbf509e4d8c82acc7b12fc6f803 (table rebalancer_extra_table_4 [id=2dc85b292eee4484aa3712fad21c8cd8]) successfully deleted
I20240501 08:14:01.702620 12973 consensus_queue.cc:237] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
W20240501 08:14:01.705063 12909 consensus_peers.cc:485] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89ec13cb2d2a418bba12b47f2ff4680b. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:01.705050 13365 raft_consensus.cc:1261] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:01.705252 13234 raft_consensus.cc:1261] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:01.705605 30511 consensus_queue.cc:1035] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:01.705953 30350 consensus_queue.cc:1035] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:01.708554 30511 raft_consensus.cc:2932] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:01.713667 13234 raft_consensus.cc:2932] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:01.713914 13365 raft_consensus.cc:2932] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:01.721017 12830 catalog_manager.cc:5484] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index -1 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:01.757102 16024 consensus_queue.cc:237] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:01.759764 13365 raft_consensus.cc:1261] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:14:01.760324 30165 consensus_queue.cc:1035] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:14:01.761124 16154 raft_consensus.cc:1261] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 08:14:01.762030 15960 consensus_peers.cc:485] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0dccc451745459d8dca19f9ebe2e47f. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:01.762084 30165 consensus_queue.cc:1035] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:14:01.764863 30532 raft_consensus.cc:2932] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:14:01.765480 16154 raft_consensus.cc:2932] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:14:01.770658 13365 raft_consensus.cc:2932] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:14:01.773726 12833 catalog_manager.cc:5484] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index -1 to 2, VOTER 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:14:02.099527 30586 ts_tablet_manager.cc:902] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:14:02.134147 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:02.145828 30586 tablet_copy_client.cc:308] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:14:02.146497 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet 89ec13cb2d2a418bba12b47f2ff4680b from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:33711)
I20240501 08:14:02.146662 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet 89ec13cb2d2a418bba12b47f2ff4680b from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:33711: session id = 5dd722c770d34d6c9154a87917117161-89ec13cb2d2a418bba12b47f2ff4680b
I20240501 08:14:02.149427 12994 tablet_copy_source_session.cc:205] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:02.153031 30586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ec13cb2d2a418bba12b47f2ff4680b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.160030 30586 tablet_copy_client.cc:783] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:02.160473 30586 tablet_copy_client.cc:647] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:02.162988 30586 tablet_copy_client.cc:523] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:02.167966 30586 tablet_bootstrap.cc:492] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:14:02.187445 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:02.236469 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:02.245291 30586 tablet_bootstrap.cc:492] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:02.245831 30586 tablet_bootstrap.cc:492] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:14:02.246204 30586 ts_tablet_manager.cc:1367] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.078s user 0.058s sys 0.007s
I20240501 08:14:02.246992 30586 raft_consensus.cc:348] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:02.247385 30586 raft_consensus.cc:729] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.273073 30586 consensus_queue.cc:260] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:02.273811 30586 ts_tablet_manager.cc:1398] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.027s user 0.021s sys 0.000s
I20240501 08:14:02.274503 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-89ec13cb2d2a418bba12b47f2ff4680b received from {username='slave'} at 127.6.56.71:33711
I20240501 08:14:02.274624 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 5dd722c770d34d6c9154a87917117161-89ec13cb2d2a418bba12b47f2ff4680b on tablet 89ec13cb2d2a418bba12b47f2ff4680b with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:14:02.339596 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:02.367717 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:02.387123 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:02.390986 30609 ts_tablet_manager.cc:902] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:14:02.438117 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:02.440776 30609 tablet_copy_client.cc:308] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:14:02.442045 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet b0dccc451745459d8dca19f9ebe2e47f from peer 6ed448d04e4646658c0de37d829ff15b ({username='slave'} at 127.6.56.67:56345)
I20240501 08:14:02.442205 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet b0dccc451745459d8dca19f9ebe2e47f from peer 6ed448d04e4646658c0de37d829ff15b at {username='slave'} at 127.6.56.67:56345: session id = 6ed448d04e4646658c0de37d829ff15b-b0dccc451745459d8dca19f9ebe2e47f
I20240501 08:14:02.445045 16045 tablet_copy_source_session.cc:205] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:02.447149 30609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0dccc451745459d8dca19f9ebe2e47f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:02.456240 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:02.457784 30609 tablet_copy_client.cc:783] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:02.458029 30609 tablet_copy_client.cc:647] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:02.468859 30609 tablet_copy_client.cc:523] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:02.472465 30609 tablet_bootstrap.cc:492] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: Bootstrap starting.
I20240501 08:14:02.488390 30609 tablet_bootstrap.cc:492] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:02.488831 30609 tablet_bootstrap.cc:492] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: Bootstrap complete.
I20240501 08:14:02.489140 30609 ts_tablet_manager.cc:1367] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.004s
I20240501 08:14:02.489799 30609 raft_consensus.cc:348] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:02.490016 30609 raft_consensus.cc:729] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Initialized, Role: FOLLOWER
I20240501 08:14:02.490621 30609 consensus_queue.cc:260] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:02.492389 30609 ts_tablet_manager.cc:1398] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:02.493562 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 6ed448d04e4646658c0de37d829ff15b-b0dccc451745459d8dca19f9ebe2e47f received from {username='slave'} at 127.6.56.67:56345
I20240501 08:14:02.493677 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 6ed448d04e4646658c0de37d829ff15b-b0dccc451745459d8dca19f9ebe2e47f on tablet b0dccc451745459d8dca19f9ebe2e47f with peer 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:02.550778 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(3b5240d5db124f409b28fbde8cf68f8a): perf score=0.033337
I20240501 08:14:02.650293 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(3b5240d5db124f409b28fbde8cf68f8a) complete. Timing: real 0.099s user 0.048s sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8811,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":2893,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":963,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8914,"threads_started":2,"wal-append.queue_time_us":303}
I20240501 08:14:02.652419 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(3b5240d5db124f409b28fbde8cf68f8a): 4921 bytes on disk
I20240501 08:14:02.653028 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(3b5240d5db124f409b28fbde8cf68f8a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 08:14:02.654937 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(a8156a46917c40e2b836a603ff70717a): perf score=0.033362
I20240501 08:14:02.656692 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(a8156a46917c40e2b836a603ff70717a): perf score=0.033362
I20240501 08:14:02.657675 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(3b5240d5db124f409b28fbde8cf68f8a): perf score=0.033370
I20240501 08:14:02.718631 16023 raft_consensus.cc:1203] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:02.726420 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(a8156a46917c40e2b836a603ff70717a) complete. Timing: real 0.071s user 0.052s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":793,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":28482,"threads_started":1,"wal-append.queue_time_us":28580}
I20240501 08:14:02.729359 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(a8156a46917c40e2b836a603ff70717a): 4581 bytes on disk
I20240501 08:14:02.730726 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(a8156a46917c40e2b836a603ff70717a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240501 08:14:02.738366 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(a8156a46917c40e2b836a603ff70717a) complete. Timing: real 0.081s user 0.027s sys 0.034s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":15746,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":4964,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":890,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":1920,"thread_start_us":17461,"threads_started":2,"wal-append.queue_time_us":1410}
I20240501 08:14:02.740252 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(db4717fa6c4e45d18702741c141f6d12): perf score=0.033380
I20240501 08:14:02.772837 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(db4717fa6c4e45d18702741c141f6d12): perf score=0.033391
I20240501 08:14:02.785969 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(3b5240d5db124f409b28fbde8cf68f8a) complete. Timing: real 0.128s user 0.056s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":14963,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":4312,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":945,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":15360,"threads_started":2,"wal-append.queue_time_us":417}
I20240501 08:14:02.790124 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(3b5240d5db124f409b28fbde8cf68f8a): 4921 bytes on disk
I20240501 08:14:02.790906 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(3b5240d5db124f409b28fbde8cf68f8a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240501 08:14:02.829514 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(db4717fa6c4e45d18702741c141f6d12) complete. Timing: real 0.089s user 0.067s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1340,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":922,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":35212,"threads_started":1,"wal-append.queue_time_us":35299}
I20240501 08:14:02.832221 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(db4717fa6c4e45d18702741c141f6d12): 4832 bytes on disk
I20240501 08:14:02.832813 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(db4717fa6c4e45d18702741c141f6d12) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 08:14:02.834743 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(a8156a46917c40e2b836a603ff70717a): 4581 bytes on disk
I20240501 08:14:02.835333 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(a8156a46917c40e2b836a603ff70717a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240501 08:14:02.837658 13234 raft_consensus.cc:1203] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:14:02.868027 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(db4717fa6c4e45d18702741c141f6d12) complete. Timing: real 0.095s user 0.039s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":333,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1313,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":762,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13715,"threads_started":2,"wal-append.queue_time_us":13592}
I20240501 08:14:02.870688 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(db4717fa6c4e45d18702741c141f6d12): 4832 bytes on disk
I20240501 08:14:02.871307 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(db4717fa6c4e45d18702741c141f6d12) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 08:14:03.165270 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:03.174407 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:03.184192 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:03.239631 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:03.251402 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:03.284307 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:03.286684 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:03.331161 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:14:03.388794 15692 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 92316.9us
I20240501 08:14:03.505367 15754 consensus_queue.cc:237] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:03.507236 12973 raft_consensus.cc:1261] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:03.507750 30698 consensus_queue.cc:1035] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:03.508893 15893 raft_consensus.cc:1261] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:03.509518 30698 consensus_queue.cc:1035] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:03.512051 30699 raft_consensus.cc:2932] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:03.512611 15893 raft_consensus.cc:2932] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:03.519964 12833 catalog_manager.cc:5484] T 3549f34428b34ea3920f0f0f20c24524 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:03.542330 12973 raft_consensus.cc:2932] T 3549f34428b34ea3920f0f0f20c24524 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:03.557495 16138 tablet_service.cc:1514] Processing DeleteTablet for tablet 3549f34428b34ea3920f0f0f20c24524 with delete_type TABLET_DATA_TOMBSTONED (TS 6cc32f6e2d49477f9a23ce6e444039fb not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:50114
I20240501 08:14:03.573557 30729 tablet_replica.cc:330] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb: stopping tablet replica
I20240501 08:14:03.576577 30729 raft_consensus.cc:2229] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:03.576925 30729 raft_consensus.cc:2258] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:03.577687 30729 ts_tablet_manager.cc:1875] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:03.596674 30729 ts_tablet_manager.cc:1888] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:14:03.599706 30729 log.cc:1196] T 3549f34428b34ea3920f0f0f20c24524 P 6cc32f6e2d49477f9a23ce6e444039fb: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/wals/3549f34428b34ea3920f0f0f20c24524
I20240501 08:14:03.600601 12817 catalog_manager.cc:4830] TS 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): tablet 3549f34428b34ea3920f0f0f20c24524 (table rebalancer_extra_table_36 [id=7175a5214616492ca05cc515dcc4d35b]) successfully deleted
I20240501 08:14:03.873301 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:03.879521 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:03.881857 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:03.883958 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:03.897552 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:03.916895 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:03.922205 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:03.948823 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:04.162086 12973 consensus_queue.cc:237] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:04.164600 13235 raft_consensus.cc:1261] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:04.164928 16024 raft_consensus.cc:1261] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:04.165510 30816 consensus_queue.cc:1035] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:04.165962 30511 consensus_queue.cc:1035] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:04.168429 30816 raft_consensus.cc:2932] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:04.168952 16024 raft_consensus.cc:2932] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:04.169427 13235 raft_consensus.cc:2932] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:04.175228 12833 catalog_manager.cc:5484] T 89ec13cb2d2a418bba12b47f2ff4680b P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index 4 to 5, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:04.196138 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet 89ec13cb2d2a418bba12b47f2ff4680b with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56180
I20240501 08:14:04.197301 30828 tablet_replica.cc:330] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:14:04.197520 30828 raft_consensus.cc:2229] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:04.197809 30828 raft_consensus.cc:2258] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:04.199261 30828 ts_tablet_manager.cc:1875] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:04.205044 30828 ts_tablet_manager.cc:1888] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 08:14:04.205161 30828 log.cc:1196] T 89ec13cb2d2a418bba12b47f2ff4680b P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/89ec13cb2d2a418bba12b47f2ff4680b
I20240501 08:14:04.205965 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet 89ec13cb2d2a418bba12b47f2ff4680b (table rebalance_test_table_15 [id=bbd038ca616f4b27bd4cb366577ea340]) successfully deleted
I20240501 08:14:04.289857 16157 consensus_queue.cc:237] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:04.292358 13363 raft_consensus.cc:1261] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:14:04.292773 15893 raft_consensus.cc:1261] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:14:04.293483 30865 consensus_queue.cc:1035] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:14:04.293828 30612 consensus_queue.cc:1035] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240501 08:14:04.294157 16093 consensus_peers.cc:485] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8c3a95d53fd43b08205d0f278841896. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:04.296542 30612 raft_consensus.cc:2932] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:04.297138 15893 raft_consensus.cc:2932] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:04.302337 13363 raft_consensus.cc:2932] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:04.303941 12826 catalog_manager.cc:5484] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:04.548504 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:04.566177 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:04.614799 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:04.661123 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:04.681936 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:04.723098 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:04.734445 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:04.735745 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:04.737327 30925 ts_tablet_manager.cc:902] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:14:04.791700 30925 tablet_copy_client.cc:308] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.72:45383
I20240501 08:14:04.792444 16178 tablet_copy_service.cc:140] P 6cc32f6e2d49477f9a23ce6e444039fb: Received BeginTabletCopySession request for tablet f8c3a95d53fd43b08205d0f278841896 from peer 76d830e199eb461c8ca689c9460ee0c2 ({username='slave'} at 127.6.56.65:33771)
I20240501 08:14:04.792608 16178 tablet_copy_service.cc:161] P 6cc32f6e2d49477f9a23ce6e444039fb: Beginning new tablet copy session on tablet f8c3a95d53fd43b08205d0f278841896 from peer 76d830e199eb461c8ca689c9460ee0c2 at {username='slave'} at 127.6.56.65:33771: session id = 76d830e199eb461c8ca689c9460ee0c2-f8c3a95d53fd43b08205d0f278841896
I20240501 08:14:04.795145 16178 tablet_copy_source_session.cc:205] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:04.796576 30925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8c3a95d53fd43b08205d0f278841896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:04.802924 30925 tablet_copy_client.cc:783] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:04.803130 30925 tablet_copy_client.cc:647] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:04.805141 30925 tablet_copy_client.cc:523] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:04.808661 30925 tablet_bootstrap.cc:492] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap starting.
I20240501 08:14:04.821638 30925 tablet_bootstrap.cc:492] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:04.822315 30925 tablet_bootstrap.cc:492] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: Bootstrap complete.
I20240501 08:14:04.822562 30925 ts_tablet_manager.cc:1367] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20240501 08:14:04.823107 30925 raft_consensus.cc:348] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:04.823276 30925 raft_consensus.cc:729] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d830e199eb461c8ca689c9460ee0c2, State: Initialized, Role: FOLLOWER
I20240501 08:14:04.823506 30925 consensus_queue.cc:260] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:04.824702 30925 ts_tablet_manager.cc:1398] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:04.825366 16178 tablet_copy_service.cc:342] P 6cc32f6e2d49477f9a23ce6e444039fb: Request end of tablet copy session 76d830e199eb461c8ca689c9460ee0c2-f8c3a95d53fd43b08205d0f278841896 received from {username='slave'} at 127.6.56.65:33771
I20240501 08:14:04.825496 16178 tablet_copy_service.cc:433] P 6cc32f6e2d49477f9a23ce6e444039fb: ending tablet copy session 76d830e199eb461c8ca689c9460ee0c2-f8c3a95d53fd43b08205d0f278841896 on tablet f8c3a95d53fd43b08205d0f278841896 with peer 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:05.287623 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:05.294107 12974 raft_consensus.cc:1203] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 08:14:05.318280 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:05.326774 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:05.364807 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:05.366801 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:05.371435 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:05.429054 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:05.456851 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:05.634346 16024 consensus_queue.cc:237] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:05.636577 13232 raft_consensus.cc:1261] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:05.636781 16158 raft_consensus.cc:1261] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:05.637116 30711 consensus_queue.cc:1035] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:05.637367 30532 consensus_queue.cc:1035] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:05.640108 30532 raft_consensus.cc:2932] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:14:05.640559 13232 raft_consensus.cc:2932] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:14:05.641255 16158 raft_consensus.cc:2932] T b0dccc451745459d8dca19f9ebe2e47f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:14:05.648036 12827 catalog_manager.cc:5484] T b0dccc451745459d8dca19f9ebe2e47f P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:14:05.663683 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet b0dccc451745459d8dca19f9ebe2e47f with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56180
I20240501 08:14:05.666746 31030 tablet_replica.cc:330] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:14:05.667145 31030 raft_consensus.cc:2229] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:05.667349 31030 raft_consensus.cc:2258] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:05.667975 31030 ts_tablet_manager.cc:1875] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:05.675868 31030 ts_tablet_manager.cc:1888] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:14:05.676021 31030 log.cc:1196] T b0dccc451745459d8dca19f9ebe2e47f P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/b0dccc451745459d8dca19f9ebe2e47f
I20240501 08:14:05.676788 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet b0dccc451745459d8dca19f9ebe2e47f (table rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]) successfully deleted
I20240501 08:14:05.919015 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:05.944365 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:05.964787 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:05.999255 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:06.006999 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:06.013270 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:06.022147 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(89ec13cb2d2a418bba12b47f2ff4680b): perf score=0.033351
W20240501 08:14:06.079620 16092 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.68:36991, user_credentials={real_user=slave}} blocked reactor thread for 44551.1us
I20240501 08:14:06.082976 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(13d9aa0c21e1405791e340cd79738256): perf score=0.033368
I20240501 08:14:06.118775 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:06.139497 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(89ec13cb2d2a418bba12b47f2ff4680b): perf score=0.033393
I20240501 08:14:06.157279 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(13d9aa0c21e1405791e340cd79738256): perf score=0.033403
I20240501 08:14:06.192396 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:06.221921 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(89ec13cb2d2a418bba12b47f2ff4680b) complete. Timing: real 0.199s user 0.034s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":13514,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":3876,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":851,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":39349,"threads_started":2,"wal-append.queue_time_us":13452}
I20240501 08:14:06.228595 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(d5179549829d451eb3a6e2083ef59b7f): perf score=0.033403
I20240501 08:14:06.261107 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: FlushMRSOp(89ec13cb2d2a418bba12b47f2ff4680b) complete. Timing: real 0.121s user 0.055s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":331,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":3915,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":842,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":19178,"threads_started":2,"wal-append.queue_time_us":19056}
I20240501 08:14:06.263851 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling UndoDeltaBlockGCOp(89ec13cb2d2a418bba12b47f2ff4680b): 4581 bytes on disk
I20240501 08:14:06.264734 12911 maintenance_manager.cc:643] P 76d830e199eb461c8ca689c9460ee0c2: UndoDeltaBlockGCOp(89ec13cb2d2a418bba12b47f2ff4680b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 08:14:06.273545 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(13d9aa0c21e1405791e340cd79738256) complete. Timing: real 0.189s user 0.031s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":13925,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1358,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":780,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":37365,"threads_started":2,"wal-append.queue_time_us":397}
I20240501 08:14:06.276849 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(13d9aa0c21e1405791e340cd79738256): 4921 bytes on disk
I20240501 08:14:06.278394 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(13d9aa0c21e1405791e340cd79738256) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240501 08:14:06.288028 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(d5179549829d451eb3a6e2083ef59b7f) complete. Timing: real 0.058s user 0.037s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1252,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6902,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":418,"threads_started":1,"wal-append.queue_time_us":514}
I20240501 08:14:06.291074 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(d5179549829d451eb3a6e2083ef59b7f): 4834 bytes on disk
I20240501 08:14:06.291764 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(d5179549829d451eb3a6e2083ef59b7f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
I20240501 08:14:06.294579 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(89ec13cb2d2a418bba12b47f2ff4680b): 4581 bytes on disk
I20240501 08:14:06.295069 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(89ec13cb2d2a418bba12b47f2ff4680b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240501 08:14:06.374353 13301 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:46839, user_credentials={real_user=slave}} blocked reactor thread for 47032.6us
I20240501 08:14:06.384553 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(13d9aa0c21e1405791e340cd79738256) complete. Timing: real 0.227s user 0.060s sys 0.045s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1781,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1694,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":759,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":86174,"threads_started":2,"wal-append.queue_time_us":71221}
I20240501 08:14:06.387435 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(13d9aa0c21e1405791e340cd79738256): 4921 bytes on disk
I20240501 08:14:06.388162 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(13d9aa0c21e1405791e340cd79738256) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240501 08:14:06.390692 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(d5179549829d451eb3a6e2083ef59b7f): perf score=0.033455
I20240501 08:14:06.446508 16157 consensus_queue.cc:237] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:06.449088 15893 raft_consensus.cc:1261] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:06.449088 12974 raft_consensus.cc:1261] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 08:14:06.449757 31094 consensus_queue.cc:1035] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:06.450054 31000 consensus_queue.cc:1035] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 08:14:06.452741 31094 raft_consensus.cc:2932] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:06.453377 15893 raft_consensus.cc:2932] T f8c3a95d53fd43b08205d0f278841896 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:06.453503 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(d5179549829d451eb3a6e2083ef59b7f) complete. Timing: real 0.063s user 0.045s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1438,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":945,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":649,"threads_started":1,"wal-append.queue_time_us":742}
I20240501 08:14:06.458078 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(d5179549829d451eb3a6e2083ef59b7f): 4834 bytes on disk
I20240501 08:14:06.458184 12831 catalog_manager.cc:5484] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New cstate: current_term: 1 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:06.458690 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(d5179549829d451eb3a6e2083ef59b7f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240501 08:14:06.473232 13343 tablet_service.cc:1514] Processing DeleteTablet for tablet f8c3a95d53fd43b08205d0f278841896 with delete_type TABLET_DATA_TOMBSTONED (TS 4ecb724cb4764a53988fde00407db4c0 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:56180
I20240501 08:14:06.475451 31134 tablet_replica.cc:330] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0: stopping tablet replica
I20240501 08:14:06.475663 31134 raft_consensus.cc:2229] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 08:14:06.475896 31134 raft_consensus.cc:2258] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 08:14:06.477663 31134 ts_tablet_manager.cc:1875] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 08:14:06.481890 12974 raft_consensus.cc:2932] T f8c3a95d53fd43b08205d0f278841896 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:06.483652 31134 ts_tablet_manager.cc:1888] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20240501 08:14:06.483791 31134 log.cc:1196] T f8c3a95d53fd43b08205d0f278841896 P 4ecb724cb4764a53988fde00407db4c0: Deleting WAL directory at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/wals/f8c3a95d53fd43b08205d0f278841896
I20240501 08:14:06.490099 12819 catalog_manager.cc:4830] TS 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): tablet f8c3a95d53fd43b08205d0f278841896 (table rebalancer_extra_table_28 [id=a03ecce65b3d46aba90c482af0c6730c]) successfully deleted
I20240501 08:14:06.867465 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:06.872681 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:06.884403 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:06.888509 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:06.956974 12953 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240501 08:14:06.962541 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:06.968065 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:07.004920 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:07.283448 16024 consensus_queue.cc:237] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:07.285883 13102 raft_consensus.cc:1261] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:07.285874 13365 raft_consensus.cc:1261] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240501 08:14:07.286136 15959 consensus_peers.cc:485] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec6d272070b140d9b918d182613008f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:07.286623 31211 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:07.287107 31211 consensus_queue.cc:1035] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:07.289924 31211 raft_consensus.cc:2932] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.290405 13102 raft_consensus.cc:2932] T ec6d272070b140d9b918d182613008f4 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.297312 13365 raft_consensus.cc:2932] T ec6d272070b140d9b918d182613008f4 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.297806 12829 catalog_manager.cc:5484] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 6 to 7, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 2 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.362263 13234 consensus_queue.cc:237] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
W20240501 08:14:07.365250 13168 consensus_peers.cc:485] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6fefd76544642ddb787cc6febe8df82. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:07.365442 16158 raft_consensus.cc:1261] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:07.365679 13365 raft_consensus.cc:1261] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:07.366058 30798 consensus_queue.cc:1035] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:07.366367 31239 consensus_queue.cc:1035] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:07.369129 31239 raft_consensus.cc:2932] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:14:07.369642 16158 raft_consensus.cc:2932] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:14:07.374509 13365 raft_consensus.cc:2932] T e6fefd76544642ddb787cc6febe8df82 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:14:07.377765 12824 catalog_manager.cc:5484] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 5 to 6, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:14:07.414113 16024 consensus_queue.cc:237] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:14:07.416149 12973 raft_consensus.cc:1261] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:07.416149 13365 raft_consensus.cc:1261] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:07.416354 15959 consensus_peers.cc:485] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc4ad07cecff43f0b867e4db444957e7. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:07.416736 30711 consensus_queue.cc:1035] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:07.417270 30711 consensus_queue.cc:1035] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:07.419961 30711 raft_consensus.cc:2932] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:14:07.420531 13365 raft_consensus.cc:2932] T fc4ad07cecff43f0b867e4db444957e7 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:14:07.425505 12973 raft_consensus.cc:2932] T fc4ad07cecff43f0b867e4db444957e7 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:14:07.428862 12825 catalog_manager.cc:5484] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 3 to 4, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:14:07.467620 13234 consensus_queue.cc:237] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:07.469869 13104 raft_consensus.cc:1261] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 08:14:07.469962 16024 raft_consensus.cc:1261] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240501 08:14:07.470227 13168 consensus_peers.cc:485] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b -> Peer 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383): Couldn't send request to peer 6cc32f6e2d49477f9a23ce6e444039fb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6aa998326f3c4ced9afbee7e54bffa0e. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:07.470430 31242 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:07.470695 30923 consensus_queue.cc:1035] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:07.481415 30923 raft_consensus.cc:2932] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:14:07.481982 13102 raft_consensus.cc:2932] T 6aa998326f3c4ced9afbee7e54bffa0e P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:14:07.487749 12824 catalog_manager.cc:5484] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b reported cstate change: config changed from index 4 to 5, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New cstate: current_term: 2 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:14:07.488116 16024 raft_consensus.cc:2932] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } }
I20240501 08:14:07.531610 12973 consensus_queue.cc:237] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:07.537585 16024 raft_consensus.cc:1261] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 08:14:07.538408 12909 consensus_peers.cc:485] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 427bc3b61a874f83bc27d9be24732e00. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:07.538401 13234 raft_consensus.cc:1261] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:07.540112 30511 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:07.540690 30511 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:07.549414 30979 raft_consensus.cc:2932] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.549880 16024 raft_consensus.cc:2932] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.569518 13234 raft_consensus.cc:2932] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.571038 12826 catalog_manager.cc:5484] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 reported cstate change: config changed from index 5 to 6, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.647920 15893 consensus_queue.cc:237] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:07.650377 15754 raft_consensus.cc:1261] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:07.650858 15826 consensus_peers.cc:485] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f305a6e24ab1436b895db037f5d72870. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:07.650861 30978 consensus_queue.cc:1035] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:07.651481 13365 raft_consensus.cc:1261] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:07.652619 30978 consensus_queue.cc:1035] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:07.655063 31092 raft_consensus.cc:2932] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:07.656000 13365 raft_consensus.cc:2932] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:07.659760 15754 raft_consensus.cc:2932] T f305a6e24ab1436b895db037f5d72870 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:07.663002 12825 catalog_manager.cc:5484] T f305a6e24ab1436b895db037f5d72870 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 3 to 4, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:07.704963 31330 ts_tablet_manager.cc:902] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:14:07.706401 31330 tablet_copy_client.cc:308] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:14:07.706449 15893 consensus_queue.cc:237] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:14:07.707121 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet ec6d272070b140d9b918d182613008f4 from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:48147)
I20240501 08:14:07.707271 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet ec6d272070b140d9b918d182613008f4 from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:48147: session id = 982f49fc87f149439cda0de2261b6bfd-ec6d272070b140d9b918d182613008f4
I20240501 08:14:07.708950 16023 raft_consensus.cc:1261] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:07.709398 13365 raft_consensus.cc:1261] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 08:14:07.709574 31319 consensus_queue.cc:1035] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:07.709977 16045 tablet_copy_source_session.cc:205] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:07.710085 31319 consensus_queue.cc:1035] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:14:07.710616 15825 consensus_peers.cc:485] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66af91fdcce84b779cf6423562b582b0. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:07.711202 31330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec6d272070b140d9b918d182613008f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:07.713580 31092 raft_consensus.cc:2932] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:14:07.714267 16024 raft_consensus.cc:2932] T 66af91fdcce84b779cf6423562b582b0 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:14:07.719636 12825 catalog_manager.cc:5484] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 reported cstate change: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:14:07.721396 31330 tablet_copy_client.cc:783] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:07.721603 31330 tablet_copy_client.cc:647] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:07.725391 31330 tablet_copy_client.cc:523] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:07.726862 13365 raft_consensus.cc:2932] T 66af91fdcce84b779cf6423562b582b0 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e3fefd4dca2d41678187047f06c10187 (127.6.56.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } }
I20240501 08:14:07.733767 31330 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:14:07.778381 13102 consensus_queue.cc:237] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:07.783177 16158 raft_consensus.cc:1261] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:07.783255 13232 raft_consensus.cc:1261] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer e3fefd4dca2d41678187047f06c10187: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 08:14:07.783870 31330 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:07.784338 31330 tablet_bootstrap.cc:492] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:14:07.784622 31330 ts_tablet_manager.cc:1367] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.051s user 0.045s sys 0.003s
I20240501 08:14:07.784574 31001 consensus_queue.cc:1035] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:07.785418 31330 raft_consensus.cc:348] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:07.785631 31330 raft_consensus.cc:729] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:14:07.785914 31330 consensus_queue.cc:260] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:07.786214 31001 consensus_queue.cc:1035] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:07.786489 31330 ts_tablet_manager.cc:1398] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:07.787706 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-ec6d272070b140d9b918d182613008f4 received from {username='slave'} at 127.6.56.69:48147
I20240501 08:14:07.787827 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-ec6d272070b140d9b918d182613008f4 on tablet ec6d272070b140d9b918d182613008f4 with peer 982f49fc87f149439cda0de2261b6bfd
W20240501 08:14:07.792666 13040 consensus_peers.cc:485] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99ecb2ec0b3d4fd7a02896cc9a28ba27. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:07.792526 31362 raft_consensus.cc:2932] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.796802 16158 raft_consensus.cc:2932] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.799010 13232 raft_consensus.cc:2932] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.801525 12826 catalog_manager.cc:5484] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187 reported cstate change: config changed from index 5 to 6, VOTER 982f49fc87f149439cda0de2261b6bfd (127.6.56.69) added. New cstate: current_term: 1 leader_uuid: "e3fefd4dca2d41678187047f06c10187" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:07.821871 31376 ts_tablet_manager.cc:902] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:14:07.825381 31376 tablet_copy_client.cc:308] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:14:07.826007 13254 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet e6fefd76544642ddb787cc6febe8df82 from peer 4d516916eeac468db514eeb928f06190 ({username='slave'} at 127.6.56.70:33497)
I20240501 08:14:07.826181 13254 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet e6fefd76544642ddb787cc6febe8df82 from peer 4d516916eeac468db514eeb928f06190 at {username='slave'} at 127.6.56.70:33497: session id = 4d516916eeac468db514eeb928f06190-e6fefd76544642ddb787cc6febe8df82
I20240501 08:14:07.829438 13254 tablet_copy_source_session.cc:205] T e6fefd76544642ddb787cc6febe8df82 P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:14:07.831306 31376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6fefd76544642ddb787cc6febe8df82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:07.837322 31376 tablet_copy_client.cc:783] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 5 data blocks...
I20240501 08:14:07.837873 15893 consensus_queue.cc:237] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:07.841086 31387 ts_tablet_manager.cc:902] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:14:07.841463 13365 raft_consensus.cc:1261] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:07.842083 31319 consensus_queue.cc:1035] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:07.842622 31387 tablet_copy_client.cc:308] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.71:36997
I20240501 08:14:07.843322 16045 tablet_copy_service.cc:140] P 5dd722c770d34d6c9154a87917117161: Received BeginTabletCopySession request for tablet fc4ad07cecff43f0b867e4db444957e7 from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:60749)
I20240501 08:14:07.843461 16045 tablet_copy_service.cc:161] P 5dd722c770d34d6c9154a87917117161: Beginning new tablet copy session on tablet fc4ad07cecff43f0b867e4db444957e7 from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:60749: session id = e3fefd4dca2d41678187047f06c10187-fc4ad07cecff43f0b867e4db444957e7
W20240501 08:14:07.844622 15825 consensus_peers.cc:485] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 -> Peer 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997): Couldn't send request to peer 5dd722c770d34d6c9154a87917117161. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9102cc8e9b92469ba2ccaa25fea2cc01. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:07.847028 16045 tablet_copy_source_session.cc:205] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:07.848999 15754 raft_consensus.cc:1261] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:07.849133 31387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4ad07cecff43f0b867e4db444957e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:07.849810 31319 consensus_queue.cc:1035] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:07.850651 31376 tablet_copy_client.cc:647] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:07.853719 31376 tablet_copy_client.cc:523] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:07.857084 31376 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: Bootstrap starting.
I20240501 08:14:07.859506 30978 raft_consensus.cc:2932] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:07.860162 13363 raft_consensus.cc:2932] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:07.868492 15754 raft_consensus.cc:2932] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:07.870926 12826 catalog_manager.cc:5484] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: config changed from index 3 to 4, VOTER 5dd722c770d34d6c9154a87917117161 (127.6.56.71) added. New cstate: current_term: 1 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:07.878245 31387 tablet_copy_client.cc:783] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:07.878473 31387 tablet_copy_client.cc:647] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:07.880023 31387 tablet_copy_client.cc:523] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:07.883899 31387 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:14:07.898494 31376 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:07.898876 31376 tablet_bootstrap.cc:492] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: Bootstrap complete.
I20240501 08:14:07.899127 31376 ts_tablet_manager.cc:1367] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: Time spent bootstrapping tablet: real 0.042s user 0.019s sys 0.004s
I20240501 08:14:07.899791 31376 raft_consensus.cc:348] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:07.900022 31376 raft_consensus.cc:729] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d516916eeac468db514eeb928f06190, State: Initialized, Role: FOLLOWER
I20240501 08:14:07.900269 31376 consensus_queue.cc:260] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:07.901000 31376 ts_tablet_manager.cc:1398] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240501 08:14:07.901655 13254 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 4d516916eeac468db514eeb928f06190-e6fefd76544642ddb787cc6febe8df82 received from {username='slave'} at 127.6.56.70:33497
I20240501 08:14:07.901801 13254 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 4d516916eeac468db514eeb928f06190-e6fefd76544642ddb787cc6febe8df82 on tablet e6fefd76544642ddb787cc6febe8df82 with peer 4d516916eeac468db514eeb928f06190
I20240501 08:14:07.912943 15939 maintenance_manager.cc:419] P 4d516916eeac468db514eeb928f06190: Scheduling UndoDeltaBlockGCOp(e6fefd76544642ddb787cc6febe8df82): 4581 bytes on disk
I20240501 08:14:07.914366 15827 maintenance_manager.cc:643] P 4d516916eeac468db514eeb928f06190: UndoDeltaBlockGCOp(e6fefd76544642ddb787cc6febe8df82) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240501 08:14:07.921748 31387 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:07.922444 31387 tablet_bootstrap.cc:492] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:14:07.922765 31387 ts_tablet_manager.cc:1367] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.018s
I20240501 08:14:07.923447 31387 raft_consensus.cc:348] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:14:07.923758 31387 raft_consensus.cc:729] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:14:07.924064 31387 consensus_queue.cc:260] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:14:07.926111 31387 ts_tablet_manager.cc:1398] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 08:14:07.930218 16045 tablet_copy_service.cc:342] P 5dd722c770d34d6c9154a87917117161: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-fc4ad07cecff43f0b867e4db444957e7 received from {username='slave'} at 127.6.56.66:60749
I20240501 08:14:07.930338 16045 tablet_copy_service.cc:433] P 5dd722c770d34d6c9154a87917117161: ending tablet copy session e3fefd4dca2d41678187047f06c10187-fc4ad07cecff43f0b867e4db444957e7 on tablet fc4ad07cecff43f0b867e4db444957e7 with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:14:07.986701 31407 ts_tablet_manager.cc:902] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: Initiating tablet copy from peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:14:07.992072 31407 tablet_copy_client.cc:308] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.67:33645
I20240501 08:14:07.992978 13254 tablet_copy_service.cc:140] P 6ed448d04e4646658c0de37d829ff15b: Received BeginTabletCopySession request for tablet 6aa998326f3c4ced9afbee7e54bffa0e from peer 6cc32f6e2d49477f9a23ce6e444039fb ({username='slave'} at 127.6.56.72:54825)
I20240501 08:14:07.993172 13254 tablet_copy_service.cc:161] P 6ed448d04e4646658c0de37d829ff15b: Beginning new tablet copy session on tablet 6aa998326f3c4ced9afbee7e54bffa0e from peer 6cc32f6e2d49477f9a23ce6e444039fb at {username='slave'} at 127.6.56.72:54825: session id = 6cc32f6e2d49477f9a23ce6e444039fb-6aa998326f3c4ced9afbee7e54bffa0e
I20240501 08:14:07.993667 31330 ts_tablet_manager.cc:902] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:14:07.996004 13254 tablet_copy_source_session.cc:205] T 6aa998326f3c4ced9afbee7e54bffa0e P 6ed448d04e4646658c0de37d829ff15b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:07.997659 31407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aa998326f3c4ced9afbee7e54bffa0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:08.019107 31407 tablet_copy_client.cc:783] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:08.019474 31407 tablet_copy_client.cc:647] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:08.021483 31407 tablet_copy_client.cc:523] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:08.027720 31330 tablet_copy_client.cc:308] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.65:46839
I20240501 08:14:08.028187 31407 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap starting.
I20240501 08:14:08.028467 12994 tablet_copy_service.cc:140] P 76d830e199eb461c8ca689c9460ee0c2: Received BeginTabletCopySession request for tablet 427bc3b61a874f83bc27d9be24732e00 from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:47087)
I20240501 08:14:08.028681 12994 tablet_copy_service.cc:161] P 76d830e199eb461c8ca689c9460ee0c2: Beginning new tablet copy session on tablet 427bc3b61a874f83bc27d9be24732e00 from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:47087: session id = 982f49fc87f149439cda0de2261b6bfd-427bc3b61a874f83bc27d9be24732e00
I20240501 08:14:08.045719 12994 tablet_copy_source_session.cc:205] T 427bc3b61a874f83bc27d9be24732e00 P 76d830e199eb461c8ca689c9460ee0c2: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:14:08.048043 31330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 427bc3b61a874f83bc27d9be24732e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:08.054225 31407 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:08.054626 31407 tablet_bootstrap.cc:492] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: Bootstrap complete.
I20240501 08:14:08.054885 31407 ts_tablet_manager.cc:1367] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.001s
I20240501 08:14:08.055538 31407 raft_consensus.cc:348] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:08.055757 31407 raft_consensus.cc:729] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Initialized, Role: FOLLOWER
I20240501 08:14:08.057488 31330 tablet_copy_client.cc:783] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 5 data blocks...
I20240501 08:14:08.057456 15754 consensus_queue.cc:237] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:08.056030 31407 consensus_queue.cc:260] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:08.065060 31407 ts_tablet_manager.cc:1398] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240501 08:14:08.066915 13254 tablet_copy_service.cc:342] P 6ed448d04e4646658c0de37d829ff15b: Request end of tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-6aa998326f3c4ced9afbee7e54bffa0e received from {username='slave'} at 127.6.56.72:54825
I20240501 08:14:08.067054 13254 tablet_copy_service.cc:433] P 6ed448d04e4646658c0de37d829ff15b: ending tablet copy session 6cc32f6e2d49477f9a23ce6e444039fb-6aa998326f3c4ced9afbee7e54bffa0e on tablet 6aa998326f3c4ced9afbee7e54bffa0e with peer 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:14:08.076238 15692 consensus_peers.cc:485] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89fc6e35197e45acb3f934888784d1aa. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:08.076257 13102 raft_consensus.cc:1261] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:14:08.076851 31299 consensus_queue.cc:1035] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:14:08.079493 16158 raft_consensus.cc:1261] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Refusing update from remote peer 982f49fc87f149439cda0de2261b6bfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:14:08.080137 31299 consensus_queue.cc:1035] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 08:14:08.092773 31330 tablet_copy_client.cc:647] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:08.093530 30827 raft_consensus.cc:2932] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:08.094218 13102 raft_consensus.cc:2932] T 89fc6e35197e45acb3f934888784d1aa P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:08.095280 31330 tablet_copy_client.cc:523] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:08.099797 16158 raft_consensus.cc:2932] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:08.101431 12825 catalog_manager.cc:5484] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd reported cstate change: config changed from index -1 to 2, VOTER 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) added. New cstate: current_term: 1 leader_uuid: "982f49fc87f149439cda0de2261b6bfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } }
I20240501 08:14:08.102998 31330 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:14:08.126427 31330 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:08.126786 31330 tablet_bootstrap.cc:492] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:14:08.127048 31330 ts_tablet_manager.cc:1367] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.003s
I20240501 08:14:08.127646 31330 raft_consensus.cc:348] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:08.127826 31330 raft_consensus.cc:729] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:14:08.128067 31330 consensus_queue.cc:260] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:08.128742 31330 ts_tablet_manager.cc:1398] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:14:08.129599 12994 tablet_copy_service.cc:342] P 76d830e199eb461c8ca689c9460ee0c2: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-427bc3b61a874f83bc27d9be24732e00 received from {username='slave'} at 127.6.56.69:47087
I20240501 08:14:08.129702 12994 tablet_copy_service.cc:433] P 76d830e199eb461c8ca689c9460ee0c2: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-427bc3b61a874f83bc27d9be24732e00 on tablet 427bc3b61a874f83bc27d9be24732e00 with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:14:08.164184 12973 consensus_queue.cc:237] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:08.166643 13232 raft_consensus.cc:1261] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 08:14:08.166702 13102 raft_consensus.cc:1261] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Refusing update from remote peer 76d830e199eb461c8ca689c9460ee0c2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 08:14:08.167115 12907 consensus_peers.cc:485] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31b3831e0f36425982b1627dd5508c99. This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:08.167129 30979 consensus_queue.cc:1035] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:08.167423 30511 consensus_queue.cc:1035] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:08.173348 30511 raft_consensus.cc:2932] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:14:08.176806 13232 raft_consensus.cc:2932] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:14:08.179769 12826 catalog_manager.cc:5484] T 31b3831e0f36425982b1627dd5508c99 P 76d830e199eb461c8ca689c9460ee0c2 reported cstate change: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New cstate: current_term: 1 leader_uuid: "76d830e199eb461c8ca689c9460ee0c2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:14:08.180002 13102 raft_consensus.cc:2932] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 4d516916eeac468db514eeb928f06190 (127.6.56.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:14:08.198999 31451 ts_tablet_manager.cc:902] T f305a6e24ab1436b895db037f5d72870 P 76d830e199eb461c8ca689c9460ee0c2: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
F20240501 08:14:08.199375 31451 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-03895995b2e5483998d442855006b24c.queue_time_us, tablet-download-pool-03895995b2e5483998d442855006b24c.queue_time_us) (tablet-download-pool-03895995b2e5483998d442855006b24c.run_cpu_time_us, tablet-download-pool-03895995b2e5483998d442855006b24c.run_cpu_time_us) (tablet-download-pool-03895995b2e5483998d442855006b24c.run_wall_time_us, tablet-download-pool-03895995b2e5483998d442855006b24c.run_wall_time_us) (tablet-download-pool-08106d7d42674f01bbee94b6236fb9f4.queue_time_us, tablet-download-pool-08106d7d42674f01bbee94b6236fb9f4.queue_time_us) (tablet-download-pool-08106d7d42674f01bbee94b6236fb9f4.run_cpu_time_us, tablet-download-pool-08106d7d42674f01bbee94b6236fb9f4.run_cpu_time_us) (tablet-download-pool-08106d7d42674f01bbee94b6236fb9f4.run_wall_time_us, tablet-download-pool-08106d7d42674f01bbee94b6236fb9f4.run_wall_time_us) (tablet-download-pool-181a5dd80e5a448b9a63d037e8ae93e7.queue_time_us, tablet-download-pool-181a5dd80e5a448b9a63d037e8ae93e7.queue_time_us) (tablet-download-pool-181a5dd80e5a448b9a63d037e8ae93e7.run_cpu_time_us, tablet-download-pool-181a5dd80e5a448b9a63d037e8ae93e7.run_cpu_time_us) (tablet-download-pool-181a5dd80e5a448b9a63d037e8ae93e7.run_wall_time_us, tablet-download-pool-181a5dd80e5a448b9a63d037e8ae93e7.run_wall_time_us) (tablet-download-pool-2cca8e4588be4791800840a638bf0372.queue_time_us, tablet-download-pool-2cca8e4588be4791800840a638bf0372.queue_time_us) (tablet-download-pool-2cca8e4588be4791800840a638bf0372.run_cpu_time_us, tablet-download-pool-2cca8e4588be4791800840a638bf0372.run_cpu_time_us) (tablet-download-pool-2cca8e4588be4791800840a638bf0372.run_wall_time_us, tablet-download-pool-2cca8e4588be4791800840a638bf0372.run_wall_time_us) (tablet-download-pool-2f795db6dcb644d691c3ba9414a62c51.queue_time_us, tablet-download-pool-2f795db6dcb644d691c3ba9414a62c51.queue_time_us) (tablet-download-pool-2f795db6dcb644d691c3ba9414a62c51.run_cpu_time_us, tablet-download-pool-2f795db6dcb644d691c3ba9414a62c51.run_cpu_time_us) (tablet-download-pool-2f795db6dcb644d691c3ba9414a62c51.run_wall_time_us, tablet-download-pool-2f795db6dcb644d691c3ba9414a62c51.run_wall_time_us) (tablet-download-pool-315d149a26c14982aae10b5fb800dc82.queue_time_us, tablet-download-pool-315d149a26c14982aae10b5fb800dc82.queue_time_us) (tablet-download-pool-315d149a26c14982aae10b5fb800dc82.run_cpu_time_us, tablet-download-pool-315d149a26c14982aae10b5fb800dc82.run_cpu_time_us) (tablet-download-pool-315d149a26c14982aae10b5fb800dc82.run_wall_time_us, tablet-download-pool-315d149a26c14982aae10b5fb800dc82.run_wall_time_us) (tablet-download-pool-3549f34428b34ea3920f0f0f20c24524.queue_time_us, tablet-download-pool-3549f34428b34ea3920f0f0f20c24524.queue_time_us) (tablet-download-pool-3549f34428b34ea3920f0f0f20c24524.run_cpu_time_us, tablet-download-pool-3549f34428b34ea3920f0f0f20c24524.run_cpu_time_us) (tablet-download-pool-3549f34428b34ea3920f0f0f20c24524.run_wall_time_us, tablet-download-pool-3549f34428b34ea3920f0f0f20c24524.run_wall_time_us) (tablet-download-pool-5e61276f6f4c40f8987747b997ac52ac.queue_time_us, tablet-download-pool-5e61276f6f4c40f8987747b997ac52ac.queue_time_us) (tablet-download-pool-5e61276f6f4c40f8987747b997ac52ac.run_cpu_time_us, tablet-download-pool-5e61276f6f4c40f8987747b997ac52ac.run_cpu_time_us) (tablet-download-pool-5e61276f6f4c40f8987747b997ac52ac.run_wall_time_us, tablet-download-pool-5e61276f6f4c40f8987747b997ac52ac.run_wall_time_us) (tablet-download-pool-7da2efbf509e4d8c82acc7b12fc6f803.queue_time_us, tablet-download-pool-7da2efbf509e4d8c82acc7b12fc6f803.queue_time_us) (tablet-download-pool-7da2efbf509e4d8c82acc7b12fc6f803.run_cpu_time_us, tablet-download-pool-7da2efbf509e4d8c82acc7b12fc6f803.run_cpu_time_us) (tablet-download-pool-7da2efbf509e4d8c82acc7b12fc6f803.run_wall_time_us, tablet-download-pool-7da2efbf509e4d8c82acc7b12fc6f803.run_wall_time_us) (tablet-download-pool-949d6765e68146d998ff4f5d2b120d68.queue_time_us, tablet-download-pool-949d6765e68146d998ff4f5d2b120d68.queue_time_us) (tablet-download-pool-949d6765e68146d998ff4f5d2b120d68.run_cpu_time_us, tablet-download-pool-949d6765e68146d998ff4f5d2b120d68.run_cpu_time_us) (tablet-download-pool-949d6765e68146d998ff4f5d2b120d68.run_wall_time_us, tablet-download-pool-949d6765e68146d998ff4f5d2b120d68.run_wall_time_us) (tablet-download-pool-a6154bf2fadb45e58c27c46165b5824f.queue_time_us, tablet-download-pool-a6154bf2fadb45e58c27c46165b5824f.queue_time_us) (tablet-download-pool-a6154bf2fadb45e58c27c46165b5824f.run_cpu_time_us, tablet-download-pool-a6154bf2fadb45e58c27c46165b5824f.run_cpu_time_us) (tablet-download-pool-a6154bf2fadb45e58c27c46165b5824f.run_wall_time_us, tablet-download-pool-a6154bf2fadb45e58c27c46165b5824f.run_wall_time_us) (tablet-download-pool-b8a350eade1c41e4b4c29e487374552f.queue_time_us, tablet-download-pool-b8a350eade1c41e4b4c29e487374552f.queue_time_us) (tablet-download-pool-b8a350eade1c41e4b4c29e487374552f.run_cpu_time_us, tablet-download-pool-b8a350eade1c41e4b4c29e487374552f.run_cpu_time_us) (tablet-download-pool-b8a350eade1c41e4b4c29e487374552f.run_wall_time_us, tablet-download-pool-b8a350eade1c41e4b4c29e487374552f.run_wall_time_us) (tablet-download-pool-bdabd94a46dc4a5796949fdf9d0c698c.queue_time_us, tablet-download-pool-bdabd94a46dc4a5796949fdf9d0c698c.queue_time_us) (tablet-download-pool-bdabd94a46dc4a5796949fdf9d0c698c.run_cpu_time_us, tablet-download-pool-bdabd94a46dc4a5796949fdf9d0c698c.run_cpu_time_us) (tablet-download-pool-bdabd94a46dc4a5796949fdf9d0c698c.run_wall_time_us, tablet-download-pool-bdabd94a46dc4a5796949fdf9d0c698c.run_wall_time_us) (tablet-download-pool-f305a6e24ab1436b895db037f5d72870.queue_time_us, tablet-download-pool-f305a6e24ab1436b895db037f5d72870.queue_time_us) (tablet-download-pool-f8c3a95d53fd43b08205d0f278841896.queue_time_us, tablet-download-pool-f8c3a95d53fd43b08205d0f278841896.queue_time_us) (tablet-download-pool-f8c3a95d53fd43b08205d0f278841896.run_cpu_time_us, tablet-download-pool-f8c3a95d53fd43b08205d0f278841896.run_cpu_time_us) (tablet-download-pool-f8c3a95d53fd43b08205d0f278841896.run_wall_time_us, tablet-download-pool-f8c3a95d53fd43b08205d0f278841896.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f5afb4fddcd google::LogMessage::Fail() at ??:0
@ 0x7f5afb501b93 google::LogMessage::SendToLog() at ??:0
I20240501 08:14:08.203766 15754 raft_consensus.cc:1203] T ec6d272070b140d9b918d182613008f4 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.6->[2.7-2.7] Dedup: 2.7->[]
@ 0x7f5afb4fd7cc google::LogMessage::Flush() at ??:0
@ 0x7f5afb4fef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5afe23e742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f5afe20af8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f5afe204cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f5b0d6578de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 08:14:08.231349 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(5741e5458ccd4af6a3972495ccf40b8f): perf score=0.033361
@ 0x7f5b0d66fdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f5b0d79a480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f5b0d7d2a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f5b0d7b5981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f5b0d7b570e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f5afe20eb75 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 08:14:08.266438 15802 maintenance_manager.cc:419] P 982f49fc87f149439cda0de2261b6bfd: Scheduling UndoDeltaBlockGCOp(427bc3b61a874f83bc27d9be24732e00): 4579 bytes on disk
@ 0x7f5afe21219c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240501 08:14:08.274030 15693 maintenance_manager.cc:643] P 982f49fc87f149439cda0de2261b6bfd: UndoDeltaBlockGCOp(427bc3b61a874f83bc27d9be24732e00) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
@ 0x7f5afe212019 std::_Function_handler<>::_M_invoke() at ??:0
I20240501 08:14:08.287737 13020 maintenance_manager.cc:419] P 76d830e199eb461c8ca689c9460ee0c2: Scheduling FlushMRSOp(74d1a1cd0bf94d0d9a87674d41838d2d): perf score=0.033378
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f5afe1ed01e kudu::Thread::SuperviseThread() at ??:0
I20240501 08:14:08.294591 31330 ts_tablet_manager.cc:902] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: Initiating tablet copy from peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:14:08.295835 31387 ts_tablet_manager.cc:902] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:14:08.305394 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(faa5b3517ffc490cab68d09bbe2d5c0e): perf score=0.033383
@ 0x7f5b015726db start_thread at ??:0
@ 0x7f5afa3b071f clone at ??:0
I20240501 08:14:08.316056 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(faa5b3517ffc490cab68d09bbe2d5c0e): perf score=0.033390
I20240501 08:14:08.349910 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: FlushMRSOp(5741e5458ccd4af6a3972495ccf40b8f) complete. Timing: real 0.118s user 0.046s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":19361,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":24234,"drs_written":1,"lbm_read_time_us":564,"lbm_reads_lt_1ms":4,"lbm_write_time_us":920,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":31762,"threads_started":2,"wal-append.queue_time_us":14731}
I20240501 08:14:08.351202 31387 tablet_copy_client.cc:308] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:14:08.351816 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 66af91fdcce84b779cf6423562b582b0 from peer e3fefd4dca2d41678187047f06c10187 ({username='slave'} at 127.6.56.66:39283)
I20240501 08:14:08.351970 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 66af91fdcce84b779cf6423562b582b0 from peer e3fefd4dca2d41678187047f06c10187 at {username='slave'} at 127.6.56.66:39283: session id = e3fefd4dca2d41678187047f06c10187-66af91fdcce84b779cf6423562b582b0
I20240501 08:14:08.353008 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling UndoDeltaBlockGCOp(5741e5458ccd4af6a3972495ccf40b8f): 4581 bytes on disk
I20240501 08:14:08.354713 15913 tablet_copy_source_session.cc:205] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:08.355484 13042 maintenance_manager.cc:643] P e3fefd4dca2d41678187047f06c10187: UndoDeltaBlockGCOp(5741e5458ccd4af6a3972495ccf40b8f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":549,"lbm_reads_lt_1ms":4}
I20240501 08:14:08.357342 31387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66af91fdcce84b779cf6423562b582b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:08.378399 15893 raft_consensus.cc:1203] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 08:14:08.393976 31330 tablet_copy_client.cc:308] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.66:35509
I20240501 08:14:08.394692 13124 tablet_copy_service.cc:140] P e3fefd4dca2d41678187047f06c10187: Received BeginTabletCopySession request for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 from peer 982f49fc87f149439cda0de2261b6bfd ({username='slave'} at 127.6.56.69:40479)
I20240501 08:14:08.394866 13124 tablet_copy_service.cc:161] P e3fefd4dca2d41678187047f06c10187: Beginning new tablet copy session on tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 from peer 982f49fc87f149439cda0de2261b6bfd at {username='slave'} at 127.6.56.69:40479: session id = 982f49fc87f149439cda0de2261b6bfd-99ecb2ec0b3d4fd7a02896cc9a28ba27
I20240501 08:14:08.395054 31387 tablet_copy_client.cc:783] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:08.395373 31387 tablet_copy_client.cc:647] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:08.398661 13124 tablet_copy_source_session.cc:205] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P e3fefd4dca2d41678187047f06c10187: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 08:14:08.399864 31387 tablet_copy_client.cc:523] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:08.402240 31330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:08.406503 13102 raft_consensus.cc:1203] T fc4ad07cecff43f0b867e4db444957e7 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:08.418509 31484 ts_tablet_manager.cc:902] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: Initiating tablet copy from peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:14:08.420790 16158 raft_consensus.cc:1203] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.4->[2.5-2.5] Dedup: 2.5->[]
I20240501 08:14:08.422875 31387 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: Bootstrap starting.
I20240501 08:14:08.427623 31330 tablet_copy_client.cc:783] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 5 data blocks...
I20240501 08:14:08.436139 31330 tablet_copy_client.cc:647] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:08.437089 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(faa5b3517ffc490cab68d09bbe2d5c0e) complete. Timing: real 0.121s user 0.052s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":451,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1205,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":912,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":9172,"threads_started":2,"wal-append.queue_time_us":330}
I20240501 08:14:08.438946 31330 tablet_copy_client.cc:523] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:08.440269 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(faa5b3517ffc490cab68d09bbe2d5c0e): 4921 bytes on disk
I20240501 08:14:08.443681 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(faa5b3517ffc490cab68d09bbe2d5c0e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240501 08:14:08.446810 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(5741e5458ccd4af6a3972495ccf40b8f): perf score=0.033423
I20240501 08:14:08.447551 31330 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap starting.
I20240501 08:14:08.453716 31387 tablet.cc:1722] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240501 08:14:08.454254 31387 tablet.cc:1620] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: MemRowSet was empty: no flush needed.
I20240501 08:14:08.454362 31387 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20240501 08:14:08.456892 31387 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:08.457293 31387 tablet_bootstrap.cc:492] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: Bootstrap complete.
I20240501 08:14:08.479454 31484 tablet_copy_client.cc:308] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: tablet copy: Beginning tablet copy session from remote peer at address 127.6.56.70:43301
I20240501 08:14:08.481129 15913 tablet_copy_service.cc:140] P 4d516916eeac468db514eeb928f06190: Received BeginTabletCopySession request for tablet 9102cc8e9b92469ba2ccaa25fea2cc01 from peer 5dd722c770d34d6c9154a87917117161 ({username='slave'} at 127.6.56.71:37437)
I20240501 08:14:08.481293 15913 tablet_copy_service.cc:161] P 4d516916eeac468db514eeb928f06190: Beginning new tablet copy session on tablet 9102cc8e9b92469ba2ccaa25fea2cc01 from peer 5dd722c770d34d6c9154a87917117161 at {username='slave'} at 127.6.56.71:37437: session id = 5dd722c770d34d6c9154a87917117161-9102cc8e9b92469ba2ccaa25fea2cc01
I20240501 08:14:08.483863 31387 ts_tablet_manager.cc:1367] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.002s
I20240501 08:14:08.484185 15913 tablet_copy_source_session.cc:205] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 08:14:08.484758 31387 raft_consensus.cc:348] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:14:08.485005 31387 raft_consensus.cc:729] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3fefd4dca2d41678187047f06c10187, State: Initialized, Role: FOLLOWER
I20240501 08:14:08.485244 31387 consensus_queue.cc:260] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:14:08.485780 31387 ts_tablet_manager.cc:1398] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:14:08.490825 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session e3fefd4dca2d41678187047f06c10187-66af91fdcce84b779cf6423562b582b0 received from {username='slave'} at 127.6.56.66:39283
I20240501 08:14:08.490967 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session e3fefd4dca2d41678187047f06c10187-66af91fdcce84b779cf6423562b582b0 on tablet 66af91fdcce84b779cf6423562b582b0 with peer e3fefd4dca2d41678187047f06c10187
I20240501 08:14:08.492946 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9102cc8e9b92469ba2ccaa25fea2cc01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:08.506065 31484 tablet_copy_client.cc:783] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 0 data blocks...
I20240501 08:14:08.509652 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(faa5b3517ffc490cab68d09bbe2d5c0e) complete. Timing: real 0.204s user 0.043s sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":286,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1444,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":863,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":8095,"threads_started":2,"wal-append.queue_time_us":317}
I20240501 08:14:08.512739 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(faa5b3517ffc490cab68d09bbe2d5c0e): 4921 bytes on disk
I20240501 08:14:08.506289 31484 tablet_copy_client.cc:647] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: tablet copy: Starting download of 1 WAL segments...
I20240501 08:14:08.513334 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(faa5b3517ffc490cab68d09bbe2d5c0e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240501 08:14:08.525159 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(74d1a1cd0bf94d0d9a87674d41838d2d): perf score=0.033439
I20240501 08:14:08.525337 31484 tablet_copy_client.cc:523] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 08:14:08.534451 31484 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: Bootstrap starting.
I20240501 08:14:08.535949 31330 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:08.537942 31330 tablet_bootstrap.cc:492] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: Bootstrap complete.
I20240501 08:14:08.540549 31330 ts_tablet_manager.cc:1367] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: Time spent bootstrapping tablet: real 0.093s user 0.017s sys 0.005s
I20240501 08:14:08.541357 31330 raft_consensus.cc:348] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:08.541574 31330 raft_consensus.cc:729] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 982f49fc87f149439cda0de2261b6bfd, State: Initialized, Role: FOLLOWER
I20240501 08:14:08.541888 31330 consensus_queue.cc:260] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:08.542547 31330 ts_tablet_manager.cc:1398] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:08.545953 13124 tablet_copy_service.cc:342] P e3fefd4dca2d41678187047f06c10187: Request end of tablet copy session 982f49fc87f149439cda0de2261b6bfd-99ecb2ec0b3d4fd7a02896cc9a28ba27 received from {username='slave'} at 127.6.56.69:40479
I20240501 08:14:08.546260 13124 tablet_copy_service.cc:433] P e3fefd4dca2d41678187047f06c10187: ending tablet copy session 982f49fc87f149439cda0de2261b6bfd-99ecb2ec0b3d4fd7a02896cc9a28ba27 on tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27 with peer 982f49fc87f149439cda0de2261b6bfd
I20240501 08:14:08.570997 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: FlushMRSOp(5741e5458ccd4af6a3972495ccf40b8f) complete. Timing: real 0.124s user 0.039s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1095,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1566,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12781,"threads_started":1,"wal-append.queue_time_us":321}
I20240501 08:14:08.574096 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling UndoDeltaBlockGCOp(5741e5458ccd4af6a3972495ccf40b8f): 4581 bytes on disk
I20240501 08:14:08.574728 13172 maintenance_manager.cc:643] P 6ed448d04e4646658c0de37d829ff15b: UndoDeltaBlockGCOp(5741e5458ccd4af6a3972495ccf40b8f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 08:14:08.587265 31484 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 08:14:08.587639 31484 tablet_bootstrap.cc:492] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: Bootstrap complete.
I20240501 08:14:08.587872 31484 ts_tablet_manager.cc:1367] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.006s
I20240501 08:14:08.588682 31484 raft_consensus.cc:348] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:08.588884 31484 raft_consensus.cc:729] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Initialized, Role: FOLLOWER
I20240501 08:14:08.589071 31484 consensus_queue.cc:260] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:08.591213 31484 ts_tablet_manager.cc:1398] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:14:08.593830 15913 tablet_copy_service.cc:342] P 4d516916eeac468db514eeb928f06190: Request end of tablet copy session 5dd722c770d34d6c9154a87917117161-9102cc8e9b92469ba2ccaa25fea2cc01 received from {username='slave'} at 127.6.56.71:37437
I20240501 08:14:08.593947 15913 tablet_copy_service.cc:433] P 4d516916eeac468db514eeb928f06190: ending tablet copy session 5dd722c770d34d6c9154a87917117161-9102cc8e9b92469ba2ccaa25fea2cc01 on tablet 9102cc8e9b92469ba2ccaa25fea2cc01 with peer 5dd722c770d34d6c9154a87917117161
I20240501 08:14:08.626693 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(74d1a1cd0bf94d0d9a87674d41838d2d) complete. Timing: real 0.101s user 0.037s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1335,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1321,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":240,"threads_started":1,"wal-append.queue_time_us":359}
I20240501 08:14:08.629485 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(74d1a1cd0bf94d0d9a87674d41838d2d): 4834 bytes on disk
I20240501 08:14:08.630089 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(74d1a1cd0bf94d0d9a87674d41838d2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240501 08:14:08.696242 15692 connection.cc:535] client connection to 127.6.56.65:46839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.696404 15960 connection.cc:535] client connection to 127.6.56.65:46839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.696569 15692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.696635 15960 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.697016 16093 connection.cc:535] client connection to 127.6.56.65:46839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.697046 15826 connection.cc:535] client connection to 127.6.56.65:46839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.697229 16093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.697252 15826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.697611 13301 connection.cc:535] client connection to 127.6.56.65:46839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.697836 13301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.698213 13041 connection.cc:535] client connection to 127.6.56.65:46839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.698334 13171 connection.cc:535] client connection to 127.6.56.65:46839 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.698388 13041 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.698519 13171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:08.709553 13041 consensus_peers.cc:485] T 8cc25959b69a41c98d196ed88117b26f P e3fefd4dca2d41678187047f06c10187 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.710031 13041 consensus_peers.cc:485] T db4717fa6c4e45d18702741c141f6d12 P e3fefd4dca2d41678187047f06c10187 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.710268 13041 consensus_peers.cc:485] T da0cf43235be4fa9bf53b1067138bc72 P e3fefd4dca2d41678187047f06c10187 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.722034 13301 consensus_peers.cc:485] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.722186 13301 consensus_peers.cc:485] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.722277 13301 consensus_peers.cc:485] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.722375 13301 consensus_peers.cc:485] T 2f795db6dcb644d691c3ba9414a62c51 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.722462 13301 consensus_peers.cc:485] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.722568 13301 consensus_peers.cc:485] T d1af72684b304cab9a7d960084ea5926 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.722668 13301 consensus_peers.cc:485] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.722766 13301 consensus_peers.cc:485] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.723950 16093 consensus_peers.cc:485] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.724181 16093 consensus_peers.cc:485] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.726589 15692 consensus_peers.cc:485] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.726897 15692 consensus_peers.cc:485] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.727027 15692 consensus_peers.cc:485] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.727130 15692 consensus_peers.cc:485] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.727209 15692 consensus_peers.cc:462] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:08.727298 15692 consensus_peers.cc:485] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.729035 15692 consensus_peers.cc:485] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.734345 15960 consensus_peers.cc:485] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.734480 15960 consensus_peers.cc:485] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.734642 15960 consensus_peers.cc:485] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.734740 15960 consensus_peers.cc:485] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.734823 15960 consensus_peers.cc:485] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.734907 15960 consensus_peers.cc:485] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.737880 15960 consensus_peers.cc:485] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.738757 15826 consensus_peers.cc:462] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:08.738868 15826 consensus_peers.cc:485] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.738981 15826 consensus_peers.cc:485] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.739174 15826 consensus_peers.cc:485] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.739895 15826 consensus_peers.cc:485] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.741114 13171 consensus_peers.cc:485] T 106da82bab514cab94c32effacf35691 P 6ed448d04e4646658c0de37d829ff15b -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.742853 13171 consensus_peers.cc:485] T 08106d7d42674f01bbee94b6236fb9f4 P 6ed448d04e4646658c0de37d829ff15b -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.748693 13171 consensus_peers.cc:485] T 4fc5dbfe3a9b4c84a3217981527e73ea P 6ed448d04e4646658c0de37d829ff15b -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.762341 15960 consensus_peers.cc:485] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:08.780642 15826 consensus_peers.cc:462] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:08.780977 15802 maintenance_manager.cc:419] P 982f49fc87f149439cda0de2261b6bfd: Scheduling UndoDeltaBlockGCOp(99ecb2ec0b3d4fd7a02896cc9a28ba27): 4581 bytes on disk
I20240501 08:14:08.781565 15693 maintenance_manager.cc:643] P 982f49fc87f149439cda0de2261b6bfd: UndoDeltaBlockGCOp(99ecb2ec0b3d4fd7a02896cc9a28ba27) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240501 08:14:08.804142 13102 raft_consensus.cc:1203] T 66af91fdcce84b779cf6423562b582b0 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 08:14:08.844847 15755 raft_consensus.cc:1203] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240501 08:14:08.999665 15692 consensus_peers.cc:462] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.007413 16023 raft_consensus.cc:1203] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 08:14:09.115047 13343 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240501 08:14:09.118052 16005 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240501 08:14:09.130429 15872 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:09.209200 16138 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 08:14:09.245286 13213 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240501 08:14:09.282368 15736 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240501 08:14:09.299072 15826 consensus_peers.cc:462] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.335588 13084 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240501 08:14:09.354125 31199 raft_consensus.cc:482] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.354326 31199 raft_consensus.cc:504] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:09.355008 31199 leader_election.cc:288] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:14:09.356770 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a5baf3631b45d68a2003fc4539f799" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:14:09.361260 31553 raft_consensus.cc:482] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.361441 31553 raft_consensus.cc:504] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:09.362191 31553 leader_election.cc:288] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:14:09.363319 13363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714a84eae9324caeb4f25e798d6c35c2" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
W20240501 08:14:09.363749 13301 leader_election.cc:334] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.363869 13301 leader_election.cc:302] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0; no voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.365132 31199 raft_consensus.cc:2726] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:09.370714 15692 leader_election.cc:334] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.370873 15692 leader_election.cc:302] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.371112 31553 raft_consensus.cc:2726] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:09.408741 30923 raft_consensus.cc:482] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.408950 30923 raft_consensus.cc:504] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.409665 30923 leader_election.cc:288] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:14:09.410423 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4b83a76d7b44039ebb17245fcd8a3c" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
W20240501 08:14:09.411195 13171 leader_election.cc:334] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.411365 13171 leader_election.cc:302] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b; no voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.411751 30923 raft_consensus.cc:2726] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:09.416031 31001 raft_consensus.cc:482] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.416208 31001 raft_consensus.cc:504] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:09.416904 31001 leader_election.cc:288] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:14:09.417470 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ba66f68a134d09bc53d840ba8191d6" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
W20240501 08:14:09.424153 13041 leader_election.cc:334] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.424319 13041 leader_election.cc:302] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3fefd4dca2d41678187047f06c10187; no voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.424559 31001 raft_consensus.cc:2726] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:09.426749 31211 raft_consensus.cc:482] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.426918 31211 raft_consensus.cc:504] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.428643 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:14:09.430078 31211 leader_election.cc:288] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
W20240501 08:14:09.433027 15960 leader_election.cc:334] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.433140 15960 leader_election.cc:302] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 6cc32f6e2d49477f9a23ce6e444039fb, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.433993 31211 raft_consensus.cc:2726] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:14:09.434625 15692 consensus_peers.cc:462] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.438800 31199 raft_consensus.cc:482] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.438943 31199 raft_consensus.cc:504] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:09.439544 31199 leader_election.cc:288] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:14:09.440255 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714a84eae9324caeb4f25e798d6c35c2" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:14:09.440501 15755 raft_consensus.cc:2454] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:14:09.440896 13300 leader_election.cc:302] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters:
W20240501 08:14:09.441241 13301 leader_election.cc:334] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.441222 31199 raft_consensus.cc:2781] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:14:09.441372 31199 raft_consensus.cc:482] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.441437 31199 raft_consensus.cc:3037] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.445676 31199 raft_consensus.cc:504] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:09.446707 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "714a84eae9324caeb4f25e798d6c35c2" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:14:09.446815 31199 leader_election.cc:288] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:14:09.446902 15755 raft_consensus.cc:3037] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:14:09.447569 13301 leader_election.cc:334] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.448968 15755 raft_consensus.cc:2454] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 2.
I20240501 08:14:09.449312 13300 leader_election.cc:302] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.449553 31199 raft_consensus.cc:2781] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:09.449705 31199 raft_consensus.cc:686] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [term 2 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:14:09.449949 31199 consensus_queue.cc:237] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:09.462181 12830 catalog_manager.cc:5484] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 1 to 2, leader changed from 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 2 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
I20240501 08:14:09.484601 31211 raft_consensus.cc:482] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.484810 31211 raft_consensus.cc:504] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.485487 31211 leader_election.cc:288] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
I20240501 08:14:09.486284 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ec13cb2d2a418bba12b47f2ff4680b" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
W20240501 08:14:09.487506 15960 leader_election.cc:334] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.487636 15960 leader_election.cc:302] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.487881 31211 raft_consensus.cc:2726] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:09.560760 31319 raft_consensus.cc:482] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.560951 31319 raft_consensus.cc:504] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:09.561710 31319 leader_election.cc:288] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:14:09.562800 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8156a46917c40e2b836a603ff70717a" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
W20240501 08:14:09.564147 15826 leader_election.cc:334] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.564272 15826 leader_election.cc:302] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.564743 31319 raft_consensus.cc:2726] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:09.565336 31553 raft_consensus.cc:482] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.565541 31553 raft_consensus.cc:504] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
W20240501 08:14:09.567606 15692 leader_election.cc:334] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.568264 31553 leader_election.cc:288] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:14:09.568369 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9a5b5333d54f6d9d8186e4d9b1b1be" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:14:09.568935 15691 leader_election.cc:302] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.569279 31553 raft_consensus.cc:2726] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:09.576427 31365 raft_consensus.cc:482] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.576678 31365 raft_consensus.cc:504] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.578233 31365 leader_election.cc:288] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:14:09.579675 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:14:09.579874 16023 raft_consensus.cc:2454] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 2.
I20240501 08:14:09.580437 16092 leader_election.cc:302] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:14:09.582580 31365 raft_consensus.cc:2781] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 08:14:09.582684 31365 raft_consensus.cc:482] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.582775 31365 raft_consensus.cc:3037] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:09.590760 31365 raft_consensus.cc:504] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.608225 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf908880298b462391a84f5a8d4e3a6b" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:14:09.608484 16023 raft_consensus.cc:3037] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:09.610577 16023 raft_consensus.cc:2454] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 3.
I20240501 08:14:09.611042 16092 leader_election.cc:302] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:14:09.611361 31558 raft_consensus.cc:2781] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:09.612463 31365 leader_election.cc:288] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
W20240501 08:14:09.615597 16093 leader_election.cc:334] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:09.617101 16093 leader_election.cc:334] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.618961 31558 raft_consensus.cc:686] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [term 3 LEADER]: Becoming Leader. State: Replica: 6cc32f6e2d49477f9a23ce6e444039fb, State: Running, Role: LEADER
I20240501 08:14:09.619227 31558 consensus_queue.cc:237] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.621083 30923 raft_consensus.cc:482] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.621315 30923 raft_consensus.cc:504] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:09.622539 30923 leader_election.cc:288] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:14:09.623430 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:14:09.623713 15755 raft_consensus.cc:2454] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
W20240501 08:14:09.623773 13171 leader_election.cc:334] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.626181 31211 raft_consensus.cc:482] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.626394 31211 raft_consensus.cc:504] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:09.627502 31211 leader_election.cc:288] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:14:09.628227 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:14:09.628468 13232 raft_consensus.cc:2454] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:14:09.629112 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:14:09.629390 15755 raft_consensus.cc:2454] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
I20240501 08:14:09.629966 15959 leader_election.cc:302] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b, 982f49fc87f149439cda0de2261b6bfd; no voters:
W20240501 08:14:09.630180 15960 leader_election.cc:334] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.630359 31211 raft_consensus.cc:2781] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:14:09.630450 31211 raft_consensus.cc:482] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.630534 31211 raft_consensus.cc:3037] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.633751 31211 raft_consensus.cc:504] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:09.634974 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:14:09.635160 15755 raft_consensus.cc:3037] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.636873 13232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:14:09.637060 13232 raft_consensus.cc:3037] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.637455 15755 raft_consensus.cc:2454] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 2.
I20240501 08:14:09.638491 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "427bc3b61a874f83bc27d9be24732e00" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:14:09.638731 16023 raft_consensus.cc:2379] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ed448d04e4646658c0de37d829ff15b in current term 2: Already voted for candidate 5dd722c770d34d6c9154a87917117161 in this term.
I20240501 08:14:09.639134 13170 leader_election.cc:302] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b, 982f49fc87f149439cda0de2261b6bfd; no voters: 5dd722c770d34d6c9154a87917117161, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.639436 13232 raft_consensus.cc:2454] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 2.
I20240501 08:14:09.639755 30923 raft_consensus.cc:2726] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:09.639766 31211 leader_election.cc:288] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:14:09.636137 12830 catalog_manager.cc:5484] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb reported cstate change: term changed from 2 to 3, leader changed from 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) to 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72). New cstate: current_term: 3 leader_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:09.642524 15960 leader_election.cc:302] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:14:09.643128 31559 raft_consensus.cc:2781] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:09.643266 31559 raft_consensus.cc:686] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [term 2 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:14:09.643534 31559 consensus_queue.cc:237] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
W20240501 08:14:09.645668 15960 leader_election.cc:334] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.657958 12830 catalog_manager.cc:5484] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 1 to 2, leader changed from 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 2 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:09.677474 30923 raft_consensus.cc:482] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.677649 30923 raft_consensus.cc:504] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:09.678586 30923 leader_election.cc:288] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:14:09.678736 13102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b3831e0f36425982b1627dd5508c99" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
W20240501 08:14:09.679152 13168 leader_election.cc:341] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Not found: Tablet not found: 31b3831e0f36425982b1627dd5508c99
I20240501 08:14:09.679654 13170 leader_election.cc:302] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6ed448d04e4646658c0de37d829ff15b; no voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:09.679934 30923 raft_consensus.cc:2726] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:09.680111 13171 leader_election.cc:334] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.680575 31362 raft_consensus.cc:482] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.680737 31362 raft_consensus.cc:504] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:09.681432 31362 leader_election.cc:288] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:14:09.681824 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b3831e0f36425982b1627dd5508c99" candidate_uuid: "e3fefd4dca2d41678187047f06c10187" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:14:09.682024 13234 raft_consensus.cc:2454] T 31b3831e0f36425982b1627dd5508c99 P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3fefd4dca2d41678187047f06c10187 in term 1.
W20240501 08:14:09.682433 13038 leader_election.cc:341] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Not found: Tablet not found: 31b3831e0f36425982b1627dd5508c99
W20240501 08:14:09.683218 13041 leader_election.cc:334] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.683341 13041 leader_election.cc:302] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e3fefd4dca2d41678187047f06c10187; no voters: 4d516916eeac468db514eeb928f06190, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.683679 31362 raft_consensus.cc:2726] T 31b3831e0f36425982b1627dd5508c99 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:09.693398 31571 raft_consensus.cc:482] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.693578 31571 raft_consensus.cc:504] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:09.694262 31571 leader_election.cc:288] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:14:09.694715 13102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ba66f68a134d09bc53d840ba8191d6" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187" is_pre_election: true
I20240501 08:14:09.694934 13102 raft_consensus.cc:2454] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:14:09.695358 15825 leader_election.cc:302] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters:
W20240501 08:14:09.695639 15826 leader_election.cc:334] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.695698 31319 raft_consensus.cc:2781] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:14:09.695910 31319 raft_consensus.cc:482] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.696069 31319 raft_consensus.cc:3037] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.699064 31319 raft_consensus.cc:504] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:09.699635 31319 leader_election.cc:288] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:14:09.700034 13102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4ba66f68a134d09bc53d840ba8191d6" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e3fefd4dca2d41678187047f06c10187"
I20240501 08:14:09.700203 13102 raft_consensus.cc:3037] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:14:09.701061 15826 leader_election.cc:334] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.702930 13102 raft_consensus.cc:2454] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 2.
I20240501 08:14:09.703363 15825 leader_election.cc:302] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.703639 31319 raft_consensus.cc:2781] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:09.703787 31319 raft_consensus.cc:686] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [term 2 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:14:09.704043 31319 consensus_queue.cc:237] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:09.708492 12830 catalog_manager.cc:5484] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 1 to 2, leader changed from 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 2 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
I20240501 08:14:09.711184 31559 raft_consensus.cc:482] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.711375 31559 raft_consensus.cc:504] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.712101 31559 leader_election.cc:288] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:14:09.712785 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4b83a76d7b44039ebb17245fcd8a3c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b" is_pre_election: true
I20240501 08:14:09.712996 13234 raft_consensus.cc:2454] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 1.
W20240501 08:14:09.713887 15960 leader_election.cc:334] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.714148 15960 leader_election.cc:302] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.714406 31559 raft_consensus.cc:2781] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:14:09.714497 31559 raft_consensus.cc:482] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.714620 31559 raft_consensus.cc:3037] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.715864 30923 raft_consensus.cc:482] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.716151 31199 raft_consensus.cc:482] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.716338 31199 raft_consensus.cc:504] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:09.716982 31199 leader_election.cc:288] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:14:09.716869 31559 raft_consensus.cc:504] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.717710 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9a5b5333d54f6d9d8186e4d9b1b1be" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:14:09.717919 15755 raft_consensus.cc:2454] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 1.
I20240501 08:14:09.718751 13300 leader_election.cc:302] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters:
I20240501 08:14:09.719012 31199 raft_consensus.cc:2781] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:14:09.719094 31199 raft_consensus.cc:482] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.719168 31199 raft_consensus.cc:3037] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:14:09.719316 13301 leader_election.cc:334] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.716022 30923 raft_consensus.cc:504] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
W20240501 08:14:09.719957 15960 leader_election.cc:334] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.720856 31559 leader_election.cc:288] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:14:09.721372 13234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4b83a76d7b44039ebb17245fcd8a3c" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "6ed448d04e4646658c0de37d829ff15b"
I20240501 08:14:09.721578 13234 raft_consensus.cc:3037] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.721979 31199 raft_consensus.cc:504] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:09.722033 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ec13cb2d2a418bba12b47f2ff4680b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:14:09.722222 16023 raft_consensus.cc:2454] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:14:09.722923 13170 leader_election.cc:302] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:14:09.723210 31242 raft_consensus.cc:2781] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:14:09.723297 31242 raft_consensus.cc:482] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.723379 31242 raft_consensus.cc:3037] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.723418 31199 leader_election.cc:288] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:14:09.724323 13234 raft_consensus.cc:2454] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 2.
W20240501 08:14:09.724331 13301 leader_election.cc:334] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.724669 15960 leader_election.cc:302] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.724910 31559 raft_consensus.cc:2781] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:09.725033 31559 raft_consensus.cc:686] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [term 2 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:14:09.725222 31559 consensus_queue.cc:237] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.725961 31242 raft_consensus.cc:504] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.726478 31242 leader_election.cc:288] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:14:09.727070 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd9a5b5333d54f6d9d8186e4d9b1b1be" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:14:09.727248 15755 raft_consensus.cc:3037] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.730031 15755 raft_consensus.cc:2454] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ecb724cb4764a53988fde00407db4c0 in term 2.
I20240501 08:14:09.730391 13300 leader_election.cc:302] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ecb724cb4764a53988fde00407db4c0, 982f49fc87f149439cda0de2261b6bfd; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.731189 31199 raft_consensus.cc:2781] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:09.731303 31199 raft_consensus.cc:686] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [term 2 LEADER]: Becoming Leader. State: Replica: 4ecb724cb4764a53988fde00407db4c0, State: Running, Role: LEADER
I20240501 08:14:09.731505 31199 consensus_queue.cc:237] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:09.732535 12830 catalog_manager.cc:5484] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 1 to 2, leader changed from 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 2 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:09.732784 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ec13cb2d2a418bba12b47f2ff4680b" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161"
I20240501 08:14:09.732950 16023 raft_consensus.cc:3037] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.733112 30923 leader_election.cc:288] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
W20240501 08:14:09.733669 13171 leader_election.cc:334] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:09.734567 13171 leader_election.cc:334] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.735255 16023 raft_consensus.cc:2454] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 2.
I20240501 08:14:09.735620 30923 raft_consensus.cc:482] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.735816 30923 raft_consensus.cc:504] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:09.738009 13170 leader_election.cc:302] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.739950 31242 raft_consensus.cc:2781] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:09.740128 31242 raft_consensus.cc:686] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [term 2 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:14:09.740491 31242 consensus_queue.cc:237] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:09.747485 12830 catalog_manager.cc:5484] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 1 to 2, leader changed from 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 2 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } }
I20240501 08:14:09.751910 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8156a46917c40e2b836a603ff70717a" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:14:09.752106 15893 raft_consensus.cc:2454] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 1.
I20240501 08:14:09.753832 13168 leader_election.cc:302] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:14:09.754912 31242 raft_consensus.cc:2781] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:14:09.755010 31242 raft_consensus.cc:482] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.755081 31242 raft_consensus.cc:3037] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 1 FOLLOWER]: Advancing to term 2
W20240501 08:14:09.756312 13171 leader_election.cc:334] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.756889 30923 leader_election.cc:288] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:14:09.757719 31242 raft_consensus.cc:504] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:09.758730 31242 leader_election.cc:288] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:14:09.759008 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8156a46917c40e2b836a603ff70717a" candidate_uuid: "6ed448d04e4646658c0de37d829ff15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190"
I20240501 08:14:09.759212 15893 raft_consensus.cc:3037] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.763145 15893 raft_consensus.cc:2454] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ed448d04e4646658c0de37d829ff15b in term 2.
I20240501 08:14:09.763669 13168 leader_election.cc:302] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 6ed448d04e4646658c0de37d829ff15b; no voters:
I20240501 08:14:09.764014 12830 catalog_manager.cc:5484] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 reported cstate change: term changed from 1 to 2, leader changed from 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) to 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68). New cstate: current_term: 2 leader_uuid: "4ecb724cb4764a53988fde00407db4c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:09.767207 31242 raft_consensus.cc:2781] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:09.767372 31242 raft_consensus.cc:686] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [term 2 LEADER]: Becoming Leader. State: Replica: 6ed448d04e4646658c0de37d829ff15b, State: Running, Role: LEADER
I20240501 08:14:09.767630 31242 consensus_queue.cc:237] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
W20240501 08:14:09.771322 13171 leader_election.cc:334] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.772135 12830 catalog_manager.cc:5484] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b reported cstate change: term changed from 1 to 2, leader changed from 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) to 6ed448d04e4646658c0de37d829ff15b (127.6.56.67). New cstate: current_term: 2 leader_uuid: "6ed448d04e4646658c0de37d829ff15b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } }
W20240501 08:14:09.776664 15826 consensus_peers.cc:462] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.786444 31319 raft_consensus.cc:482] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.786599 31319 raft_consensus.cc:504] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:09.787166 31319 leader_election.cc:288] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:14:09.787607 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a5baf3631b45d68a2003fc4539f799" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
I20240501 08:14:09.787818 13365 raft_consensus.cc:2454] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:14:09.788389 15825 leader_election.cc:302] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters:
W20240501 08:14:09.788574 15826 leader_election.cc:334] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.788693 31319 raft_consensus.cc:2781] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:14:09.788797 31319 raft_consensus.cc:482] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76d830e199eb461c8ca689c9460ee0c2)
I20240501 08:14:09.788894 31319 raft_consensus.cc:3037] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.791523 31319 raft_consensus.cc:504] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:09.792127 31319 leader_election.cc:288] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 election: Requested vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
W20240501 08:14:09.793449 15826 leader_election.cc:334] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:09.794476 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34a5baf3631b45d68a2003fc4539f799" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0"
I20240501 08:14:09.794657 13365 raft_consensus.cc:3037] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:09.797731 13365 raft_consensus.cc:2454] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 2.
I20240501 08:14:09.798128 15825 leader_election.cc:302] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 4ecb724cb4764a53988fde00407db4c0; no voters: 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:09.798616 31319 raft_consensus.cc:2781] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:09.799546 31319 raft_consensus.cc:686] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [term 2 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:14:09.799809 31319 consensus_queue.cc:237] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:09.813683 12830 catalog_manager.cc:5484] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 1 to 2, leader changed from 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65) to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 2 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } }
W20240501 08:14:09.925107 13301 consensus_peers.cc:485] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 2
Maximum Replica Count | 43
Average Replica Count | 21.750000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 1
Maximum | 3
Average | 2.850000
stderr: W20240501 08:12:13.999778 16210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:12:14.000231 16210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:12:14.009513 16210 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
E20240501 08:12:14.621692 16308 ksck_remote.cc:634] unable to open table rebalance_test_table_2: Not found: the table does not exist: table_name: "rebalance_test_table_2"
I20240501 08:12:15.231114 16210 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 08:12:16.134989 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21a198e8f040480a9343868c04aa14a6: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:16.135743 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21a198e8f040480a9343868c04aa14a6: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:16.136165 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ea3df3f0972a46c29ab991f3e578eb71: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:16.136794 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bbd038ca616f4b27bd4cb366577ea340: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:16.136919 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6c085de9eb3e44d3a1100a754cacad29: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:16.137022 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21a198e8f040480a9343868c04aa14a6: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:16.137126 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc6dee4abd8f47a9bf045445762a8869: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:16.137346 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0980a0d51532489eafb86f16ede323ef: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:16.137509 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c3bf94d0aef44991ba373437c141baf6: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:16.137733 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9d3f2a7a052c418bbe2d3bb437760b4f: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:16.137835 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table e5090602461741aebf39ef301b19e3a2: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:16.138341 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21a198e8f040480a9343868c04aa14a6: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:16.138485 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table aa780586626d49fe8f25ab6348b6cdac: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:16.138599 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 59c66af77cf24d538a3b224c075adb1f: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:16.138729 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a04eed287ab4f3ead90a7aab3a14938: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:16.138836 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e717b4ed14d4b91858e6cdd752f1662: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:16.139024 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0f0bc268a64d4bee90527ed42efaf03d: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:16.139219 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ceb56f187844e50969e5632fde934fc: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:16.139321 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e5940bd1584cea8c5869059eedc2aa: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:16.139430 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ab73f40ad68f450ca7d155c31949855f: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:16.139555 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table aa780586626d49fe8f25ab6348b6cdac: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:16.139674 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6c085de9eb3e44d3a1100a754cacad29: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:16.139755 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 21a198e8f040480a9343868c04aa14a6: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:16.139864 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 59c66af77cf24d538a3b224c075adb1f: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:16.139976 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ea3df3f0972a46c29ab991f3e578eb71: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:16.140115 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc6dee4abd8f47a9bf045445762a8869: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:16.140211 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0980a0d51532489eafb86f16ede323ef: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
I20240501 08:12:16.261610 16210 rebalancer_tool.cc:1074] tablet a54efcd41aeb4c24adfa51bb543d9661: '6ed448d04e4646658c0de37d829ff15b' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:12:16.331560 16210 rebalancer_tool.cc:1074] tablet 911cc47749484e88882c0efa75fc6396: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:12:16.437022 16210 rebalancer_tool.cc:1074] tablet ba940b0d8b064284bfc71952801d17c3: 'e3fefd4dca2d41678187047f06c10187' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:12:16.508508 16210 rebalancer_tool.cc:1074] tablet 3b5240d5db124f409b28fbde8cf68f8a: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:12:16.801838 16210 rebalancer_tool.cc:1084] tablet 3720cbc21425459d9f0cb91cf825e521: '6ed448d04e4646658c0de37d829ff15b' -> '4d516916eeac468db514eeb928f06190' move ignored: Not found: 3720cbc21425459d9f0cb91cf825e521: tablet not found
I20240501 08:12:16.802063 16210 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:18.258252 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c3bf94d0aef44991ba373437c141baf6: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:18.258522 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c3bf94d0aef44991ba373437c141baf6: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:18.258711 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ea3df3f0972a46c29ab991f3e578eb71: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:18.259091 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ab73f40ad68f450ca7d155c31949855f: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:18.259313 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc6dee4abd8f47a9bf045445762a8869: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:18.259406 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table aa780586626d49fe8f25ab6348b6cdac: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:18.259938 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c3bf94d0aef44991ba373437c141baf6: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:18.260044 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e717b4ed14d4b91858e6cdd752f1662: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:18.260159 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 970c6326810a49e99e77ee988599335e: could not find any suitable replica to move from server 4d516916eeac468db514eeb928f06190 to server 4ecb724cb4764a53988fde00407db4c0
W20240501 08:12:18.260354 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6c085de9eb3e44d3a1100a754cacad29: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:18.260463 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9d3f2a7a052c418bbe2d3bb437760b4f: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:18.260551 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0980a0d51532489eafb86f16ede323ef: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:18.260644 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e5940bd1584cea8c5869059eedc2aa: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:18.260844 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a04eed287ab4f3ead90a7aab3a14938: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:18.261312 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table dc6dee4abd8f47a9bf045445762a8869: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:18.261495 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c3bf94d0aef44991ba373437c141baf6: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:18.261849 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bbd038ca616f4b27bd4cb366577ea340: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:18.261948 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 2ceb56f187844e50969e5632fde934fc: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:18.262066 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a04eed287ab4f3ead90a7aab3a14938: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:18.262208 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6c085de9eb3e44d3a1100a754cacad29: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 4d516916eeac468db514eeb928f06190
I20240501 08:12:18.335937 16210 rebalancer_tool.cc:1074] tablet 4fc5dbfe3a9b4c84a3217981527e73ea: 'e3fefd4dca2d41678187047f06c10187' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:12:18.386111 16210 rebalancer_tool.cc:1074] tablet cd9a5b5333d54f6d9d8186e4d9b1b1be: '6ed448d04e4646658c0de37d829ff15b' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:12:18.445295 16210 rebalancer_tool.cc:1074] tablet d5179549829d451eb3a6e2083ef59b7f: '4ecb724cb4764a53988fde00407db4c0' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:12:18.449615 16210 rebalancer_tool.cc:1084] tablet 47ee02938eee4f1ba72279b6e0b6774c: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move ignored: Not found: 47ee02938eee4f1ba72279b6e0b6774c: tablet not found
I20240501 08:12:18.449748 16210 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:18.740650 16859 meta_cache.cc:1015] Not found: LookupRpc { table: 'rebalance_test_table_18', partition-key: (<start>), attempt: 1 } failed: table rebalance_test_table_18 was deleted: Table deleted at 2024-05-01 08:12:18 UTC
W20240501 08:12:19.461123 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ea3df3f0972a46c29ab991f3e578eb71: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:19.461444 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e5940bd1584cea8c5869059eedc2aa: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:19.461630 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9d3f2a7a052c418bbe2d3bb437760b4f: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:19.461738 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a04eed287ab4f3ead90a7aab3a14938: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:19.462028 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6c085de9eb3e44d3a1100a754cacad29: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:19.462204 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e717b4ed14d4b91858e6cdd752f1662: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:19.462302 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0980a0d51532489eafb86f16ede323ef: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:19.462468 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table aa780586626d49fe8f25ab6348b6cdac: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:19.462576 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bbd038ca616f4b27bd4cb366577ea340: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:19.462690 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ab73f40ad68f450ca7d155c31949855f: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:19.462877 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e717b4ed14d4b91858e6cdd752f1662: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 5dd722c770d34d6c9154a87917117161
I20240501 08:12:19.561409 16210 rebalancer_tool.cc:1074] tablet 9348009ae1444e73b6120161309dc4a2: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:12:19.635717 16210 rebalancer_tool.cc:1074] tablet 427bc3b61a874f83bc27d9be24732e00: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:12:19.704900 16210 rebalancer_tool.cc:1074] tablet d1af72684b304cab9a7d960084ea5926: '6ed448d04e4646658c0de37d829ff15b' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:12:19.839107 16210 rebalancer_tool.cc:1074] tablet faa5b3517ffc490cab68d09bbe2d5c0e: 'e3fefd4dca2d41678187047f06c10187' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:12:20.232344 16210 rebalancer_tool.cc:1074] tablet 7883772f69cd4a18a2a6d568adbdeb61: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:12:20.385084 16210 rebalancer_tool.cc:1074] tablet 7c4b83a76d7b44039ebb17245fcd8a3c: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:12:20.637537 16210 rebalancer_tool.cc:1074] tablet 8a48ba4666c14848b2b5a1e62f16357b: 'e3fefd4dca2d41678187047f06c10187' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:12:21.012569 16210 rebalancer_tool.cc:1074] tablet bdabd94a46dc4a5796949fdf9d0c698c: '4d516916eeac468db514eeb928f06190' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:12:21.017127 16210 rebalancer_tool.cc:1084] tablet 4bfb69f10ef84a0ab5e448a301f952f5: '982f49fc87f149439cda0de2261b6bfd' -> '6cc32f6e2d49477f9a23ce6e444039fb' move ignored: Not found: 4bfb69f10ef84a0ab5e448a301f952f5: tablet not found
I20240501 08:12:21.017246 16210 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:21.730227 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5ec25bc622944a14a3f9a0ce15683b4f: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 6ed448d04e4646658c0de37d829ff15b
W20240501 08:12:21.730893 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0980a0d51532489eafb86f16ede323ef: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:21.731081 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5ec25bc622944a14a3f9a0ce15683b4f: could not find any suitable replica to move from server 982f49fc87f149439cda0de2261b6bfd to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:21.731232 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6c085de9eb3e44d3a1100a754cacad29: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:21.731379 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ea3df3f0972a46c29ab991f3e578eb71: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:21.731504 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a04eed287ab4f3ead90a7aab3a14938: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:21.731706 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e717b4ed14d4b91858e6cdd752f1662: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:21.731798 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table aa780586626d49fe8f25ab6348b6cdac: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:21.731985 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bbd038ca616f4b27bd4cb366577ea340: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:21.732117 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ab73f40ad68f450ca7d155c31949855f: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:21.732226 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e5940bd1584cea8c5869059eedc2aa: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:21.732343 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9d3f2a7a052c418bbe2d3bb437760b4f: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:12:21.784183 16210 rebalancer_tool.cc:1074] tablet 6c2c8548c31343ff9426fe7eda194703: '6cc32f6e2d49477f9a23ce6e444039fb' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:12:22.115619 16210 rebalancer_tool.cc:1074] tablet 94e7fb526fbc4ebc94875c00b49ac43d: '76d830e199eb461c8ca689c9460ee0c2' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:12:22.182448 16210 rebalancer_tool.cc:1074] tablet d4ba66f68a134d09bc53d840ba8191d6: '4ecb724cb4764a53988fde00407db4c0' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:12:22.235384 16210 rebalancer_tool.cc:1074] tablet 82e08735b5db41dcbccec46aa7278530: '6ed448d04e4646658c0de37d829ff15b' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:12:22.307987 16210 rebalancer_tool.cc:1074] tablet 74d6cce81d164722afad1935d4695374: 'e3fefd4dca2d41678187047f06c10187' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:12:26.389328 16210 rebalancer_tool.cc:1123] tablet ba940b0d8b064284bfc71952801d17c3: 'e3fefd4dca2d41678187047f06c10187' -> '982f49fc87f149439cda0de2261b6bfd' move is abandoned: Not found: ba940b0d8b064284bfc71952801d17c3: tablet not found
E20240501 08:12:30.540253 18524 ksck_remote.cc:634] unable to open table rebalancer_dynamic_table: Not found: the table does not exist: table_name: "rebalancer_dynamic_table"
W20240501 08:12:34.374596 18931 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.6.56.67:33645, user_credentials={real_user=slave}} blocked reactor thread for 54866.4us
I20240501 08:12:44.329195 16210 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 08:12:45.078284 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ea3df3f0972a46c29ab991f3e578eb71: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:45.078547 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table aa780586626d49fe8f25ab6348b6cdac: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:45.078725 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9d3f2a7a052c418bbe2d3bb437760b4f: could not find any suitable replica to move from server 4ecb724cb4764a53988fde00407db4c0 to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:45.079021 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ea038a9919c4a9fb5bf8203c2bf1e1d: could not find any suitable replica to move from server 5dd722c770d34d6c9154a87917117161 to server e3fefd4dca2d41678187047f06c10187
W20240501 08:12:45.079128 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5a04eed287ab4f3ead90a7aab3a14938: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:45.079203 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 16e5940bd1584cea8c5869059eedc2aa: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:45.079284 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6c085de9eb3e44d3a1100a754cacad29: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:45.079377 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table be8478913ef642319e7d9cfac923eac4: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:45.079442 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0e717b4ed14d4b91858e6cdd752f1662: could not find any suitable replica to move from server 6ed448d04e4646658c0de37d829ff15b to server 4d516916eeac468db514eeb928f06190
W20240501 08:12:45.079567 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table bbd038ca616f4b27bd4cb366577ea340: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:45.079658 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table aa62935a584c4c33ad4e2e98639049fc: could not find any suitable replica to move from server 4d516916eeac468db514eeb928f06190 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:45.079767 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 0980a0d51532489eafb86f16ede323ef: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:45.079851 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9d3f2a7a052c418bbe2d3bb437760b4f: could not find any suitable replica to move from server 76d830e199eb461c8ca689c9460ee0c2 to server 5dd722c770d34d6c9154a87917117161
W20240501 08:12:45.080106 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table ab73f40ad68f450ca7d155c31949855f: could not find any suitable replica to move from server e3fefd4dca2d41678187047f06c10187 to server 982f49fc87f149439cda0de2261b6bfd
W20240501 08:12:45.080176 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table d149afe167ad401ab5a28770ee66cca1: could not find any suitable replica to move from server 4d516916eeac468db514eeb928f06190 to server 76d830e199eb461c8ca689c9460ee0c2
W20240501 08:12:45.080338 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 48397a21eba44b8592fc40090375f0dd: could not find any suitable replica to move from server 5dd722c770d34d6c9154a87917117161 to server 6cc32f6e2d49477f9a23ce6e444039fb
W20240501 08:12:45.080430 16210 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 970c6326810a49e99e77ee988599335e: could not find any suitable replica to move from server 982f49fc87f149439cda0de2261b6bfd to server 4ecb724cb4764a53988fde00407db4c0
I20240501 08:12:45.145334 16210 rebalancer_tool.cc:1074] tablet 8c487b2c7fe24515a8970a07883c126f: '982f49fc87f149439cda0de2261b6bfd' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:12:46.301730 16210 rebalancer_tool.cc:1136] tablet 94e7fb526fbc4ebc94875c00b49ac43d: '76d830e199eb461c8ca689c9460ee0c2' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:12:46.324086 16210 rebalancer_tool.cc:1136] tablet cd9a5b5333d54f6d9d8186e4d9b1b1be: '6ed448d04e4646658c0de37d829ff15b' -> '982f49fc87f149439cda0de2261b6bfd' move completed: OK
I20240501 08:12:46.336206 16210 rebalancer_tool.cc:1136] tablet 4fc5dbfe3a9b4c84a3217981527e73ea: 'e3fefd4dca2d41678187047f06c10187' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:12:46.345463 16210 rebalancer_tool.cc:1136] tablet 3b5240d5db124f409b28fbde8cf68f8a: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:12:46.356508 16210 rebalancer_tool.cc:1136] tablet bdabd94a46dc4a5796949fdf9d0c698c: '4d516916eeac468db514eeb928f06190' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:12:46.366636 16210 rebalancer_tool.cc:1136] tablet a54efcd41aeb4c24adfa51bb543d9661: '6ed448d04e4646658c0de37d829ff15b' -> '4d516916eeac468db514eeb928f06190' move completed: OK
I20240501 08:12:46.375602 16210 rebalancer_tool.cc:1136] tablet 8a48ba4666c14848b2b5a1e62f16357b: 'e3fefd4dca2d41678187047f06c10187' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:12:47.134222 16210 rebalancer_tool.cc:1136] tablet 74d6cce81d164722afad1935d4695374: 'e3fefd4dca2d41678187047f06c10187' -> '982f49fc87f149439cda0de2261b6bfd' move completed: OK
I20240501 08:12:47.142756 16210 rebalancer_tool.cc:1136] tablet 911cc47749484e88882c0efa75fc6396: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:12:47.154327 16210 rebalancer_tool.cc:1136] tablet 9348009ae1444e73b6120161309dc4a2: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:12:47.165410 16210 rebalancer_tool.cc:1136] tablet 427bc3b61a874f83bc27d9be24732e00: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:12:47.174654 16210 rebalancer_tool.cc:1136] tablet 6c2c8548c31343ff9426fe7eda194703: '6cc32f6e2d49477f9a23ce6e444039fb' -> '982f49fc87f149439cda0de2261b6bfd' move completed: OK
I20240501 08:12:47.184756 16210 rebalancer_tool.cc:1136] tablet faa5b3517ffc490cab68d09bbe2d5c0e: 'e3fefd4dca2d41678187047f06c10187' -> '982f49fc87f149439cda0de2261b6bfd' move completed: OK
I20240501 08:12:47.196439 16210 rebalancer_tool.cc:1136] tablet 82e08735b5db41dcbccec46aa7278530: '6ed448d04e4646658c0de37d829ff15b' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:12:47.215391 16210 rebalancer_tool.cc:1136] tablet 7883772f69cd4a18a2a6d568adbdeb61: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:12:47.239989 16210 rebalancer_tool.cc:1136] tablet d1af72684b304cab9a7d960084ea5926: '6ed448d04e4646658c0de37d829ff15b' -> '4d516916eeac468db514eeb928f06190' move completed: OK
I20240501 08:12:47.250046 16210 rebalancer_tool.cc:1136] tablet 7c4b83a76d7b44039ebb17245fcd8a3c: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:12:47.296680 16210 rebalancer_tool.cc:1074] tablet e6fefd76544642ddb787cc6febe8df82: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:12:47.362763 16210 rebalancer_tool.cc:1074] tablet ec6d272070b140d9b918d182613008f4: '6ed448d04e4646658c0de37d829ff15b' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:12:47.416874 16210 rebalancer_tool.cc:1074] tablet 8953bcf510a2413cbceb4924f651b771: '982f49fc87f149439cda0de2261b6bfd' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:12:47.465435 16210 rebalancer_tool.cc:1074] tablet 13d9aa0c21e1405791e340cd79738256: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:12:47.569924 16210 rebalancer_tool.cc:1074] tablet 0f723c6321424530bcd41b9ecdcbe0b3: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:12:47.626719 16210 rebalancer_tool.cc:1074] tablet fc4ad07cecff43f0b867e4db444957e7: '4d516916eeac468db514eeb928f06190' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:12:47.689164 16210 rebalancer_tool.cc:1074] tablet 6aa998326f3c4ced9afbee7e54bffa0e: '982f49fc87f149439cda0de2261b6bfd' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:12:47.742992 16210 rebalancer_tool.cc:1074] tablet 53872527ad264dd9b4b1627ff8aa46db: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:12:47.792196 16210 rebalancer_tool.cc:1074] tablet 37b0020baddc4f47b625edc2e1180e51: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:12:47.880776 16210 rebalancer_tool.cc:1074] tablet d241e087e754485cb80080754d27aef7: '4d516916eeac468db514eeb928f06190' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:12:48.601948 16210 rebalancer_tool.cc:1074] tablet b8a350eade1c41e4b4c29e487374552f: '982f49fc87f149439cda0de2261b6bfd' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:12:48.667722 16210 rebalancer_tool.cc:1074] tablet 3d16de310e7148768967be392b7af2ee: '6cc32f6e2d49477f9a23ce6e444039fb' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:12:48.743602 16210 rebalancer_tool.cc:1074] tablet fa10a6f53f7f48cf9410c80e1056439d: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:12:48.811271 16210 rebalancer_tool.cc:1074] tablet a8156a46917c40e2b836a603ff70717a: '4ecb724cb4764a53988fde00407db4c0' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:12:48.860766 16210 rebalancer_tool.cc:1074] tablet 03895995b2e5483998d442855006b24c: '982f49fc87f149439cda0de2261b6bfd' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:12:48.953277 16210 rebalancer_tool.cc:1074] tablet 7a71b0688ca04973a46b3cbc2ce680d2: '6cc32f6e2d49477f9a23ce6e444039fb' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:12:49.031162 16210 rebalancer_tool.cc:1074] tablet 321bf6596ff24ec7af79e0510eadedce: '5dd722c770d34d6c9154a87917117161' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:13:02.071662 16210 rebalancer_tool.cc:1136] tablet d4ba66f68a134d09bc53d840ba8191d6: '4ecb724cb4764a53988fde00407db4c0' -> '4d516916eeac468db514eeb928f06190' move completed: OK
I20240501 08:13:02.725947 16210 rebalancer_tool.cc:1136] tablet d5179549829d451eb3a6e2083ef59b7f: '4ecb724cb4764a53988fde00407db4c0' -> '4d516916eeac468db514eeb928f06190' move completed: OK
I20240501 08:13:03.350104 16210 rebalancer_tool.cc:1136] tablet 7a71b0688ca04973a46b3cbc2ce680d2: '6cc32f6e2d49477f9a23ce6e444039fb' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:03.363015 16210 rebalancer_tool.cc:1136] tablet 03895995b2e5483998d442855006b24c: '982f49fc87f149439cda0de2261b6bfd' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:13:03.371590 16210 rebalancer_tool.cc:1136] tablet a8156a46917c40e2b836a603ff70717a: '4ecb724cb4764a53988fde00407db4c0' -> '4d516916eeac468db514eeb928f06190' move completed: OK
I20240501 08:13:03.380949 16210 rebalancer_tool.cc:1136] tablet b8a350eade1c41e4b4c29e487374552f: '982f49fc87f149439cda0de2261b6bfd' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:13:03.392460 16210 rebalancer_tool.cc:1136] tablet 3d16de310e7148768967be392b7af2ee: '6cc32f6e2d49477f9a23ce6e444039fb' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:03.403395 16210 rebalancer_tool.cc:1136] tablet d241e087e754485cb80080754d27aef7: '4d516916eeac468db514eeb928f06190' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:03.414770 16210 rebalancer_tool.cc:1136] tablet 37b0020baddc4f47b625edc2e1180e51: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:03.424728 16210 rebalancer_tool.cc:1136] tablet 53872527ad264dd9b4b1627ff8aa46db: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:13:03.433954 16210 rebalancer_tool.cc:1136] tablet fc4ad07cecff43f0b867e4db444957e7: '4d516916eeac468db514eeb928f06190' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:03.445914 16210 rebalancer_tool.cc:1136] tablet 0f723c6321424530bcd41b9ecdcbe0b3: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:03.455279 16210 rebalancer_tool.cc:1136] tablet 13d9aa0c21e1405791e340cd79738256: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:13:03.464361 16210 rebalancer_tool.cc:1136] tablet 8953bcf510a2413cbceb4924f651b771: '982f49fc87f149439cda0de2261b6bfd' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:04.742413 16210 rebalancer_tool.cc:1136] tablet e6fefd76544642ddb787cc6febe8df82: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:13:04.761858 16210 rebalancer_tool.cc:1136] tablet 8c487b2c7fe24515a8970a07883c126f: '982f49fc87f149439cda0de2261b6bfd' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:04.772392 16210 rebalancer_tool.cc:1136] tablet fa10a6f53f7f48cf9410c80e1056439d: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:04.820449 16210 rebalancer_tool.cc:1074] tablet a6154bf2fadb45e58c27c46165b5824f: '4d516916eeac468db514eeb928f06190' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:13:04.873883 16210 rebalancer_tool.cc:1074] tablet 5741e5458ccd4af6a3972495ccf40b8f: '4ecb724cb4764a53988fde00407db4c0' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:13:05.015820 16210 rebalancer_tool.cc:1074] tablet c99eddd74b7740dcab6b93ca437b51ae: '6ed448d04e4646658c0de37d829ff15b' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:13:05.076635 16210 rebalancer_tool.cc:1074] tablet a412b8ae0aa648d7979fac58b50d86a6: '76d830e199eb461c8ca689c9460ee0c2' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:13:05.130185 16210 rebalancer_tool.cc:1074] tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27: '4ecb724cb4764a53988fde00407db4c0' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:13:05.182760 16210 rebalancer_tool.cc:1074] tablet f8203d209d624d2695e5b32bbfa0edf7: '5dd722c770d34d6c9154a87917117161' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:13:05.247157 16210 rebalancer_tool.cc:1074] tablet 5e61276f6f4c40f8987747b997ac52ac: '6ed448d04e4646658c0de37d829ff15b' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:13:05.317236 16210 rebalancer_tool.cc:1074] tablet 85056837a567420a8b480b34b6017b40: '982f49fc87f149439cda0de2261b6bfd' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:05.405586 16210 rebalancer_tool.cc:1074] tablet cabfd31e8ee941cf9f69bc04a0510d11: '4d516916eeac468db514eeb928f06190' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:13:05.503636 16210 rebalancer_tool.cc:1074] tablet 8cc25959b69a41c98d196ed88117b26f: '6cc32f6e2d49477f9a23ce6e444039fb' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:13:05.794936 16210 rebalancer_tool.cc:1074] tablet 08106d7d42674f01bbee94b6236fb9f4: 'e3fefd4dca2d41678187047f06c10187' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:13:05.879945 16210 rebalancer_tool.cc:1074] tablet 34a5baf3631b45d68a2003fc4539f799: '982f49fc87f149439cda0de2261b6bfd' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:06.074119 16210 rebalancer_tool.cc:1074] tablet ba08236c71df4c8190f6b7644ed944ed: '5dd722c770d34d6c9154a87917117161' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:13:06.149253 16210 rebalancer_tool.cc:1074] tablet f22fc6797cb24451b78a5c1e5f6306ba: '6cc32f6e2d49477f9a23ce6e444039fb' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:13:06.252110 16210 rebalancer_tool.cc:1074] tablet 181a5dd80e5a448b9a63d037e8ae93e7: 'e3fefd4dca2d41678187047f06c10187' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:13:06.371634 16210 rebalancer_tool.cc:1074] tablet 66af91fdcce84b779cf6423562b582b0: '982f49fc87f149439cda0de2261b6bfd' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
W20240501 08:13:06.766533 23350 meta_cache.cc:1261] Time spent looking up entry by key: real 0.058s user 0.000s sys 0.000s
I20240501 08:13:17.096041 16210 rebalancer_tool.cc:1136] tablet 321bf6596ff24ec7af79e0510eadedce: '5dd722c770d34d6c9154a87917117161' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:18.023749 16210 rebalancer_tool.cc:1136] tablet ec6d272070b140d9b918d182613008f4: '6ed448d04e4646658c0de37d829ff15b' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:13:18.056800 16210 rebalancer_tool.cc:1136] tablet 6aa998326f3c4ced9afbee7e54bffa0e: '982f49fc87f149439cda0de2261b6bfd' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:18.104575 16210 rebalancer_tool.cc:1074] tablet 4e330f6ac7074134ba06effe5ececcde: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:13:18.178304 16210 rebalancer_tool.cc:1074] tablet 74d1a1cd0bf94d0d9a87674d41838d2d: 'e3fefd4dca2d41678187047f06c10187' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:13:18.247934 16210 rebalancer_tool.cc:1074] tablet f305a6e24ab1436b895db037f5d72870: '6cc32f6e2d49477f9a23ce6e444039fb' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:13:20.983311 16210 rebalancer_tool.cc:1136] tablet 34a5baf3631b45d68a2003fc4539f799: '982f49fc87f149439cda0de2261b6bfd' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:21.003124 16210 rebalancer_tool.cc:1136] tablet 08106d7d42674f01bbee94b6236fb9f4: 'e3fefd4dca2d41678187047f06c10187' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:13:21.617957 16210 rebalancer_tool.cc:1136] tablet cabfd31e8ee941cf9f69bc04a0510d11: '4d516916eeac468db514eeb928f06190' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:13:21.630602 16210 rebalancer_tool.cc:1136] tablet 85056837a567420a8b480b34b6017b40: '982f49fc87f149439cda0de2261b6bfd' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:21.640266 16210 rebalancer_tool.cc:1136] tablet 5e61276f6f4c40f8987747b997ac52ac: '6ed448d04e4646658c0de37d829ff15b' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:13:21.649950 16210 rebalancer_tool.cc:1136] tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27: '4ecb724cb4764a53988fde00407db4c0' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:13:21.659699 16210 rebalancer_tool.cc:1136] tablet f22fc6797cb24451b78a5c1e5f6306ba: '6cc32f6e2d49477f9a23ce6e444039fb' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:21.669803 16210 rebalancer_tool.cc:1136] tablet ba08236c71df4c8190f6b7644ed944ed: '5dd722c770d34d6c9154a87917117161' -> '4d516916eeac468db514eeb928f06190' move completed: OK
I20240501 08:13:21.679679 16210 rebalancer_tool.cc:1136] tablet a412b8ae0aa648d7979fac58b50d86a6: '76d830e199eb461c8ca689c9460ee0c2' -> '4d516916eeac468db514eeb928f06190' move completed: OK
I20240501 08:13:21.689857 16210 rebalancer_tool.cc:1136] tablet 66af91fdcce84b779cf6423562b582b0: '982f49fc87f149439cda0de2261b6bfd' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:21.699792 16210 rebalancer_tool.cc:1136] tablet c99eddd74b7740dcab6b93ca437b51ae: '6ed448d04e4646658c0de37d829ff15b' -> '982f49fc87f149439cda0de2261b6bfd' move completed: OK
I20240501 08:13:21.712970 16210 rebalancer_tool.cc:1136] tablet 5741e5458ccd4af6a3972495ccf40b8f: '4ecb724cb4764a53988fde00407db4c0' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:13:21.723471 16210 rebalancer_tool.cc:1136] tablet a6154bf2fadb45e58c27c46165b5824f: '4d516916eeac468db514eeb928f06190' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:13:21.734290 16210 rebalancer_tool.cc:1136] tablet f8203d209d624d2695e5b32bbfa0edf7: '5dd722c770d34d6c9154a87917117161' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:21.780095 16210 rebalancer_tool.cc:1074] tablet 6fb95e3405ae45d3bfe71fb9d552e3fe: '4ecb724cb4764a53988fde00407db4c0' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:13:21.842320 16210 rebalancer_tool.cc:1074] tablet b2e7c8c1795d4c6d997c75e63dee6760: '76d830e199eb461c8ca689c9460ee0c2' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:13:21.952128 16210 rebalancer_tool.cc:1074] tablet b000ec8b190644f2883e5f824463b762: '5dd722c770d34d6c9154a87917117161' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:22.033914 16210 rebalancer_tool.cc:1074] tablet 7a98388bf2fd402b8064d2cca06f5b8b: '6ed448d04e4646658c0de37d829ff15b' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:13:22.094383 16210 rebalancer_tool.cc:1074] tablet 949d6765e68146d998ff4f5d2b120d68: 'e3fefd4dca2d41678187047f06c10187' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:13:22.146041 16210 rebalancer_tool.cc:1074] tablet 1eac68af4bc3473487bf91670cb48223: '982f49fc87f149439cda0de2261b6bfd' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:13:22.196813 16210 rebalancer_tool.cc:1074] tablet 65a3d07808414c50a330da007c8f5e19: '4d516916eeac468db514eeb928f06190' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:13:22.317160 16210 rebalancer_tool.cc:1074] tablet 3e62b4a817174374a1ffdf9f8b25fb44: '6cc32f6e2d49477f9a23ce6e444039fb' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:22.472151 16210 rebalancer_tool.cc:1074] tablet b2b0c752d9614f2fbd3863d92df609a4: '4d516916eeac468db514eeb928f06190' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:13:22.763482 16210 rebalancer_tool.cc:1074] tablet 9102cc8e9b92469ba2ccaa25fea2cc01: '76d830e199eb461c8ca689c9460ee0c2' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:22.818048 16210 rebalancer_tool.cc:1074] tablet 39b12a1d835d428085d4a24233d78c38: '982f49fc87f149439cda0de2261b6bfd' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:13:22.904052 16210 rebalancer_tool.cc:1074] tablet 6658ae0ccbcc4baf8853ab8778049030: '6ed448d04e4646658c0de37d829ff15b' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:13:23.018258 16210 rebalancer_tool.cc:1074] tablet 3c8ec0d302644d378c54e4871f85159a: '4d516916eeac468db514eeb928f06190' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:13:23.110724 16210 rebalancer_tool.cc:1074] tablet 69d21bf5577f4b449a691ef6234376d2: '76d830e199eb461c8ca689c9460ee0c2' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:23.218874 16210 rebalancer_tool.cc:1074] tablet cf908880298b462391a84f5a8d4e3a6b: '982f49fc87f149439cda0de2261b6bfd' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:13:35.186650 16210 rebalancer_tool.cc:1136] tablet 74d1a1cd0bf94d0d9a87674d41838d2d: 'e3fefd4dca2d41678187047f06c10187' -> '4d516916eeac468db514eeb928f06190' move completed: OK
I20240501 08:13:35.819089 16210 rebalancer_tool.cc:1136] tablet 4e330f6ac7074134ba06effe5ececcde: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:35.831852 16210 rebalancer_tool.cc:1136] tablet 181a5dd80e5a448b9a63d037e8ae93e7: 'e3fefd4dca2d41678187047f06c10187' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:13:35.842252 16210 rebalancer_tool.cc:1136] tablet 8cc25959b69a41c98d196ed88117b26f: '6cc32f6e2d49477f9a23ce6e444039fb' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:35.916373 16210 rebalancer_tool.cc:1074] tablet ca6339e48a2f4b9e94e32e05c7bf343b: '6ed448d04e4646658c0de37d829ff15b' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:13:36.032845 16210 rebalancer_tool.cc:1074] tablet 279b5f4d8fba4c12beec040c6ed39992: '4d516916eeac468db514eeb928f06190' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:13:36.094525 16210 rebalancer_tool.cc:1074] tablet 21719a43fa6845f8a3a5337dbe770f60: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:13:36.160558 16210 rebalancer_tool.cc:1074] tablet a328773ef57444cb8147524107b15dba: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
W20240501 08:13:37.345880 27434 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.6.56.70:43301, user_credentials={real_user=slave}} blocked reactor thread for 65167.7us
I20240501 08:13:39.784543 16210 rebalancer_tool.cc:1136] tablet 39b12a1d835d428085d4a24233d78c38: '982f49fc87f149439cda0de2261b6bfd' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:13:39.799212 16210 rebalancer_tool.cc:1136] tablet b2b0c752d9614f2fbd3863d92df609a4: '4d516916eeac468db514eeb928f06190' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:39.811215 16210 rebalancer_tool.cc:1136] tablet 3e62b4a817174374a1ffdf9f8b25fb44: '6cc32f6e2d49477f9a23ce6e444039fb' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:39.821547 16210 rebalancer_tool.cc:1136] tablet 65a3d07808414c50a330da007c8f5e19: '4d516916eeac468db514eeb928f06190' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:39.831141 16210 rebalancer_tool.cc:1136] tablet 3c8ec0d302644d378c54e4871f85159a: '4d516916eeac468db514eeb928f06190' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:39.842250 16210 rebalancer_tool.cc:1136] tablet 1eac68af4bc3473487bf91670cb48223: '982f49fc87f149439cda0de2261b6bfd' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:13:39.856052 16210 rebalancer_tool.cc:1136] tablet 9102cc8e9b92469ba2ccaa25fea2cc01: '76d830e199eb461c8ca689c9460ee0c2' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:40.684190 16210 rebalancer_tool.cc:1136] tablet 7a98388bf2fd402b8064d2cca06f5b8b: '6ed448d04e4646658c0de37d829ff15b' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:13:40.695492 16210 rebalancer_tool.cc:1136] tablet b000ec8b190644f2883e5f824463b762: '5dd722c770d34d6c9154a87917117161' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:41.328554 16210 rebalancer_tool.cc:1136] tablet b2e7c8c1795d4c6d997c75e63dee6760: '76d830e199eb461c8ca689c9460ee0c2' -> '4d516916eeac468db514eeb928f06190' move completed: OK
I20240501 08:13:41.342135 16210 rebalancer_tool.cc:1136] tablet 6fb95e3405ae45d3bfe71fb9d552e3fe: '4ecb724cb4764a53988fde00407db4c0' -> '982f49fc87f149439cda0de2261b6bfd' move completed: OK
I20240501 08:13:41.354827 16210 rebalancer_tool.cc:1136] tablet f305a6e24ab1436b895db037f5d72870: '6cc32f6e2d49477f9a23ce6e444039fb' -> '982f49fc87f149439cda0de2261b6bfd' move completed: OK
I20240501 08:13:41.365433 16210 rebalancer_tool.cc:1136] tablet 6658ae0ccbcc4baf8853ab8778049030: '6ed448d04e4646658c0de37d829ff15b' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:13:41.382037 16210 rebalancer_tool.cc:1136] tablet 69d21bf5577f4b449a691ef6234376d2: '76d830e199eb461c8ca689c9460ee0c2' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:41.431277 16210 rebalancer_tool.cc:1074] tablet 714a84eae9324caeb4f25e798d6c35c2: '6cc32f6e2d49477f9a23ce6e444039fb' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:41.489218 16210 rebalancer_tool.cc:1074] tablet b320e884c36d461ba5f2941d9d76185f: '5dd722c770d34d6c9154a87917117161' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:13:41.645426 16210 rebalancer_tool.cc:1074] tablet a713c94c6bb14d8793ee429e52f0e7a0: '4ecb724cb4764a53988fde00407db4c0' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:13:41.708359 16210 rebalancer_tool.cc:1074] tablet 315d149a26c14982aae10b5fb800dc82: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:13:41.764593 16210 rebalancer_tool.cc:1074] tablet 80ef1222e2324309a009a657ca704795: '982f49fc87f149439cda0de2261b6bfd' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:13:41.817207 16210 rebalancer_tool.cc:1074] tablet 691f5a8c1d334f6ba6464c9596db49ed: '4ecb724cb4764a53988fde00407db4c0' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:13:41.878882 16210 rebalancer_tool.cc:1074] tablet 1ba04c9b0d0a4933a123dbeb5b50145e: '5dd722c770d34d6c9154a87917117161' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:42.012260 16210 rebalancer_tool.cc:1074] tablet 6840d246ddfa42b3b8065d7136a36f5a: '76d830e199eb461c8ca689c9460ee0c2' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:13:42.085757 16210 rebalancer_tool.cc:1074] tablet 4345eabfdb894def8d7063773dc5fac6: '6cc32f6e2d49477f9a23ce6e444039fb' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:52.534852 16210 rebalancer_tool.cc:1136] tablet 21719a43fa6845f8a3a5337dbe770f60: '5dd722c770d34d6c9154a87917117161' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:52.554993 16210 rebalancer_tool.cc:1136] tablet 279b5f4d8fba4c12beec040c6ed39992: '4d516916eeac468db514eeb928f06190' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:53.261080 16210 rebalancer_tool.cc:1136] tablet 949d6765e68146d998ff4f5d2b120d68: 'e3fefd4dca2d41678187047f06c10187' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:13:53.280184 16210 rebalancer_tool.cc:1136] tablet ca6339e48a2f4b9e94e32e05c7bf343b: '6ed448d04e4646658c0de37d829ff15b' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:53.833950 16210 rebalancer_tool.cc:1136] tablet 6840d246ddfa42b3b8065d7136a36f5a: '76d830e199eb461c8ca689c9460ee0c2' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:13:53.855628 16210 rebalancer_tool.cc:1136] tablet 1ba04c9b0d0a4933a123dbeb5b50145e: '5dd722c770d34d6c9154a87917117161' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:53.865141 16210 rebalancer_tool.cc:1136] tablet 691f5a8c1d334f6ba6464c9596db49ed: '4ecb724cb4764a53988fde00407db4c0' -> 'e3fefd4dca2d41678187047f06c10187' move completed: OK
I20240501 08:13:53.877563 16210 rebalancer_tool.cc:1136] tablet 80ef1222e2324309a009a657ca704795: '982f49fc87f149439cda0de2261b6bfd' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:13:53.887564 16210 rebalancer_tool.cc:1136] tablet 315d149a26c14982aae10b5fb800dc82: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:13:55.462574 16210 rebalancer_tool.cc:1136] tablet 714a84eae9324caeb4f25e798d6c35c2: '6cc32f6e2d49477f9a23ce6e444039fb' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:13:55.476012 16210 rebalancer_tool.cc:1136] tablet a328773ef57444cb8147524107b15dba: '76d830e199eb461c8ca689c9460ee0c2' -> '6cc32f6e2d49477f9a23ce6e444039fb' move completed: OK
I20240501 08:13:55.485312 16210 rebalancer_tool.cc:1136] tablet cf908880298b462391a84f5a8d4e3a6b: '982f49fc87f149439cda0de2261b6bfd' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:13:55.593968 16210 rebalancer_tool.cc:1074] tablet 7da2efbf509e4d8c82acc7b12fc6f803: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:13:55.674279 16210 rebalancer_tool.cc:1074] tablet 608bc35666234d2cbcb816439d050196: '6ed448d04e4646658c0de37d829ff15b' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:55.726258 16210 rebalancer_tool.cc:1074] tablet db4717fa6c4e45d18702741c141f6d12: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:13:55.795728 16210 rebalancer_tool.cc:1074] tablet 2f795db6dcb644d691c3ba9414a62c51: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:13:55.848685 16210 rebalancer_tool.cc:1074] tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57: '6ed448d04e4646658c0de37d829ff15b' -> '4ecb724cb4764a53988fde00407db4c0' move scheduled
I20240501 08:13:55.906359 16210 rebalancer_tool.cc:1074] tablet 2cca8e4588be4791800840a638bf0372: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:13:56.024855 16210 rebalancer_tool.cc:1074] tablet 3549f34428b34ea3920f0f0f20c24524: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
W20240501 08:13:56.074158 29798 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.6.56.69:42897, user_credentials={real_user=slave}} blocked reactor thread for 51828.8us
I20240501 08:14:01.632735 16210 rebalancer_tool.cc:1136] tablet 4345eabfdb894def8d7063773dc5fac6: '6cc32f6e2d49477f9a23ce6e444039fb' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:14:01.657078 16210 rebalancer_tool.cc:1136] tablet a713c94c6bb14d8793ee429e52f0e7a0: '4ecb724cb4764a53988fde00407db4c0' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:14:01.666985 16210 rebalancer_tool.cc:1136] tablet b320e884c36d461ba5f2941d9d76185f: '5dd722c770d34d6c9154a87917117161' -> '982f49fc87f149439cda0de2261b6bfd' move completed: OK
I20240501 08:14:01.715298 16210 rebalancer_tool.cc:1074] tablet 89ec13cb2d2a418bba12b47f2ff4680b: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:14:01.778524 16210 rebalancer_tool.cc:1074] tablet b0dccc451745459d8dca19f9ebe2e47f: '4ecb724cb4764a53988fde00407db4c0' -> '6ed448d04e4646658c0de37d829ff15b' move scheduled
I20240501 08:14:03.584478 16210 rebalancer_tool.cc:1136] tablet 2cca8e4588be4791800840a638bf0372: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:14:04.197443 16210 rebalancer_tool.cc:1136] tablet b5fda95d9e9d48e7a4d6eb4d9b85fb57: '6ed448d04e4646658c0de37d829ff15b' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:14:04.209527 16210 rebalancer_tool.cc:1136] tablet 2f795db6dcb644d691c3ba9414a62c51: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:14:04.222213 16210 rebalancer_tool.cc:1136] tablet db4717fa6c4e45d18702741c141f6d12: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:14:04.235080 16210 rebalancer_tool.cc:1136] tablet 608bc35666234d2cbcb816439d050196: '6ed448d04e4646658c0de37d829ff15b' -> '4ecb724cb4764a53988fde00407db4c0' move completed: OK
I20240501 08:14:04.261361 16210 rebalancer_tool.cc:1136] tablet 7da2efbf509e4d8c82acc7b12fc6f803: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:14:04.304500 16210 rebalancer_tool.cc:1074] tablet f8c3a95d53fd43b08205d0f278841896: '4ecb724cb4764a53988fde00407db4c0' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:14:05.668438 16210 rebalancer_tool.cc:1136] tablet 3549f34428b34ea3920f0f0f20c24524: '6cc32f6e2d49477f9a23ce6e444039fb' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:14:05.681766 16210 rebalancer_tool.cc:1136] tablet 89ec13cb2d2a418bba12b47f2ff4680b: '4ecb724cb4764a53988fde00407db4c0' -> '5dd722c770d34d6c9154a87917117161' move completed: OK
I20240501 08:14:06.495162 16210 rebalancer_tool.cc:1136] tablet b0dccc451745459d8dca19f9ebe2e47f: '4ecb724cb4764a53988fde00407db4c0' -> '6ed448d04e4646658c0de37d829ff15b' move completed: OK
I20240501 08:14:06.507148 16210 rebalancer_tool.cc:1136] tablet f8c3a95d53fd43b08205d0f278841896: '4ecb724cb4764a53988fde00407db4c0' -> '76d830e199eb461c8ca689c9460ee0c2' move completed: OK
I20240501 08:14:07.304627 16210 rebalancer_tool.cc:1074] tablet ec6d272070b140d9b918d182613008f4: '4ecb724cb4764a53988fde00407db4c0' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:14:07.376484 16210 rebalancer_tool.cc:1074] tablet e6fefd76544642ddb787cc6febe8df82: '6ed448d04e4646658c0de37d829ff15b' -> '4d516916eeac468db514eeb928f06190' move scheduled
I20240501 08:14:07.427683 16210 rebalancer_tool.cc:1074] tablet fc4ad07cecff43f0b867e4db444957e7: '76d830e199eb461c8ca689c9460ee0c2' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:14:07.497262 16210 rebalancer_tool.cc:1074] tablet 6aa998326f3c4ced9afbee7e54bffa0e: '5dd722c770d34d6c9154a87917117161' -> '6cc32f6e2d49477f9a23ce6e444039fb' move scheduled
I20240501 08:14:07.559324 16210 rebalancer_tool.cc:1074] tablet 427bc3b61a874f83bc27d9be24732e00: '6ed448d04e4646658c0de37d829ff15b' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:14:07.664839 16210 rebalancer_tool.cc:1074] tablet f305a6e24ab1436b895db037f5d72870: '4d516916eeac468db514eeb928f06190' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:14:07.718143 16210 rebalancer_tool.cc:1074] tablet 66af91fdcce84b779cf6423562b582b0: '5dd722c770d34d6c9154a87917117161' -> 'e3fefd4dca2d41678187047f06c10187' move scheduled
I20240501 08:14:07.804438 16210 rebalancer_tool.cc:1074] tablet 99ecb2ec0b3d4fd7a02896cc9a28ba27: '6ed448d04e4646658c0de37d829ff15b' -> '982f49fc87f149439cda0de2261b6bfd' move scheduled
I20240501 08:14:07.865520 16210 rebalancer_tool.cc:1074] tablet 9102cc8e9b92469ba2ccaa25fea2cc01: '4d516916eeac468db514eeb928f06190' -> '5dd722c770d34d6c9154a87917117161' move scheduled
I20240501 08:14:08.099404 16210 rebalancer_tool.cc:1074] tablet 89fc6e35197e45acb3f934888784d1aa: 'e3fefd4dca2d41678187047f06c10187' -> '76d830e199eb461c8ca689c9460ee0c2' move scheduled
I20240501 08:14:08.178812 16210 rebalancer_tool.cc:1074] tablet 31b3831e0f36425982b1627dd5508c99: '6ed448d04e4646658c0de37d829ff15b' -> '4d516916eeac468db514eeb928f06190' move scheduled
W20240501 08:14:08.698645 31477 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:14:08.332710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.56.65:46839 (local address 127.0.0.1:42742)
0501 08:14:08.334148 (+ 1438us) negotiation.cc:107] Waiting for socket to connect
0501 08:14:08.334168 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0501 08:14:08.334298 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:14:08.697209 (+362911us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1315,"thread_start_us":730,"threads_started":1}
I20240501 08:14:08.699970 16210 rebalancer_tool.cc:1084] tablet d4ba66f68a134d09bc53d840ba8191d6: 'e3fefd4dca2d41678187047f06c10187' -> '982f49fc87f149439cda0de2261b6bfd' move ignored: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:14:08.701150 16210 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): unacceptable health status UNAVAILABLE
I20240501 08:14:10.036624 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 13023
I20240501 08:14:10.045356 13144 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:14:10.045908 15692 consensus_peers.cc:462] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:10.062358 15755 raft_consensus.cc:1261] T 427bc3b61a874f83bc27d9be24732e00 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:10.063102 31073 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:10.075961 15755 raft_consensus.cc:1261] T 714a84eae9324caeb4f25e798d6c35c2 P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:10.076874 31199 consensus_queue.cc:1035] T 714a84eae9324caeb4f25e798d6c35c2 P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:10.145053 16023 raft_consensus.cc:1261] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:10.145980 30923 consensus_queue.cc:1035] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:10.150440 16024 raft_consensus.cc:1261] T cf908880298b462391a84f5a8d4e3a6b P 5dd722c770d34d6c9154a87917117161 [term 3 FOLLOWER]: Refusing update from remote peer 6cc32f6e2d49477f9a23ce6e444039fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 08:14:10.151353 31558 consensus_queue.cc:1035] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 08:14:10.165937 13171 consensus_peers.cc:485] T 89ec13cb2d2a418bba12b47f2ff4680b P 6ed448d04e4646658c0de37d829ff15b -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.178164 16093 consensus_peers.cc:485] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:10.190235 13234 raft_consensus.cc:1261] T 7c4b83a76d7b44039ebb17245fcd8a3c P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:10.190814 31559 consensus_queue.cc:1035] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:10.191123 15892 raft_consensus.cc:1261] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Refusing update from remote peer 6ed448d04e4646658c0de37d829ff15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:10.191697 31242 consensus_queue.cc:1035] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:14:10.200531 15960 consensus_peers.cc:485] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:10.201776 13232 raft_consensus.cc:1261] T 427bc3b61a874f83bc27d9be24732e00 P 6ed448d04e4646658c0de37d829ff15b [term 2 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 08:14:10.207937 31073 consensus_queue.cc:1035] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 08:14:10.210667 15960 consensus_peers.cc:485] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.212604 13171 consensus_peers.cc:485] T a8156a46917c40e2b836a603ff70717a P 6ed448d04e4646658c0de37d829ff15b -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:10.231014 13365 raft_consensus.cc:1261] T 34a5baf3631b45d68a2003fc4539f799 P 4ecb724cb4764a53988fde00407db4c0 [term 2 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:10.231721 31571 consensus_queue.cc:1035] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 08:14:10.238849 15826 consensus_peers.cc:485] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.270535 15826 consensus_peers.cc:485] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:10.293818 15755 raft_consensus.cc:1261] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Refusing update from remote peer 4ecb724cb4764a53988fde00407db4c0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:10.294449 31598 consensus_queue.cc:1035] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 08:14:10.300052 13301 consensus_peers.cc:485] T cd9a5b5333d54f6d9d8186e4d9b1b1be P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.330222 15826 consensus_peers.cc:462] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:10.482925 13151 maintenance_manager.cc:419] P e3fefd4dca2d41678187047f06c10187: Scheduling FlushMRSOp(d4ba66f68a134d09bc53d840ba8191d6): perf score=0.033354
I20240501 08:14:10.498236 13099 raft_consensus.cc:1261] T d4ba66f68a134d09bc53d840ba8191d6 P e3fefd4dca2d41678187047f06c10187 [term 2 FOLLOWER]: Refusing update from remote peer 4d516916eeac468db514eeb928f06190: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 08:14:10.501060 31571 consensus_queue.cc:1035] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:10.514338 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 13023
W20240501 08:14:10.516669 15960 consensus_peers.cc:485] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.576402 15690 connection.cc:535] server connection from 127.6.56.66:33955 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:10.576665 15959 connection.cc:535] client connection to 127.6.56.66:35509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:10.577164 15691 connection.cc:535] client connection to 127.6.56.66:35509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:14:10.577337 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 13153
W20240501 08:14:10.577389 15825 connection.cc:535] client connection to 127.6.56.66:35509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:10.578208 13299 connection.cc:535] server connection from 127.6.56.66:41029 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:10.578886 15691 consensus_peers.cc:485] T 55b77e87381d402ab836985100ccb279 P 982f49fc87f149439cda0de2261b6bfd -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.579035 15691 consensus_peers.cc:485] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.579116 15691 consensus_peers.cc:485] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.579187 15691 consensus_peers.cc:485] T b320e884c36d461ba5f2941d9d76185f P 982f49fc87f149439cda0de2261b6bfd -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.579275 15691 consensus_peers.cc:485] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.580564 15825 consensus_peers.cc:485] T d4ba66f68a134d09bc53d840ba8191d6 P 4d516916eeac468db514eeb928f06190 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.580729 15825 consensus_peers.cc:485] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.581140 15959 consensus_peers.cc:485] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.581291 15959 consensus_peers.cc:485] T 8953bcf510a2413cbceb4924f651b771 P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.581403 15959 consensus_peers.cc:485] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.581503 15959 consensus_peers.cc:485] T 3c8ec0d302644d378c54e4871f85159a P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.581605 15959 consensus_peers.cc:485] T f755002e2f824db7993f912d234295bb P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.581710 15959 consensus_peers.cc:485] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.581815 15959 consensus_peers.cc:485] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.581930 15959 consensus_peers.cc:485] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.582178 15691 consensus_peers.cc:485] T 691f5a8c1d334f6ba6464c9596db49ed P 982f49fc87f149439cda0de2261b6bfd -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.587168 13170 connection.cc:535] client connection to 127.6.56.66:35509 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:10.589164 13170 consensus_peers.cc:485] T 53872527ad264dd9b4b1627ff8aa46db P 6ed448d04e4646658c0de37d829ff15b -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.589298 13170 consensus_peers.cc:485] T b2b0c752d9614f2fbd3863d92df609a4 P 6ed448d04e4646658c0de37d829ff15b -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.592475 15825 consensus_peers.cc:485] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:10.600142 13274 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:14:10.654376 15825 consensus_peers.cc:485] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.655877 15692 consensus_peers.cc:462] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:10.659406 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling FlushMRSOp(7883772f69cd4a18a2a6d568adbdeb61): perf score=0.033396
W20240501 08:14:10.671010 13301 consensus_peers.cc:485] T 7a98388bf2fd402b8064d2cca06f5b8b P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:14:10.689112 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: FlushMRSOp(7883772f69cd4a18a2a6d568adbdeb61) complete. Timing: real 0.029s user 0.023s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":364,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":2008,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":835,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":743,"threads_started":2,"wal-append.queue_time_us":551}
I20240501 08:14:10.691659 13411 maintenance_manager.cc:419] P 4ecb724cb4764a53988fde00407db4c0: Scheduling UndoDeltaBlockGCOp(7883772f69cd4a18a2a6d568adbdeb61): 4581 bytes on disk
I20240501 08:14:10.692206 13302 maintenance_manager.cc:643] P 4ecb724cb4764a53988fde00407db4c0: UndoDeltaBlockGCOp(7883772f69cd4a18a2a6d568adbdeb61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240501 08:14:10.722268 15825 consensus_peers.cc:485] T 29335562faf0492f9acf2848fc582dd8 P 4d516916eeac468db514eeb928f06190 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.722865 15825 consensus_peers.cc:485] T 6c5c7f2d0156405da42d25d1e18ccbfa P 4d516916eeac468db514eeb928f06190 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.751178 15826 consensus_peers.cc:462] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:10.764972 13301 consensus_peers.cc:485] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.772523 15960 consensus_peers.cc:485] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.774554 15692 consensus_peers.cc:485] T 321bf6596ff24ec7af79e0510eadedce P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.787335 13300 consensus_peers.cc:485] T a328773ef57444cb8147524107b15dba P 4ecb724cb4764a53988fde00407db4c0 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.787884 15960 consensus_peers.cc:485] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.798483 13300 consensus_peers.cc:485] T 3d16de310e7148768967be392b7af2ee P 4ecb724cb4764a53988fde00407db4c0 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.809866 15826 consensus_peers.cc:485] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.811077 15692 consensus_peers.cc:485] T 3549f34428b34ea3920f0f0f20c24524 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.823935 13301 consensus_peers.cc:485] T 7da2efbf509e4d8c82acc7b12fc6f803 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.853273 13300 consensus_peers.cc:485] T d93c796748404ca096aa600c94d5d5de P 4ecb724cb4764a53988fde00407db4c0 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.892192 15692 consensus_peers.cc:485] T 2cca8e4588be4791800840a638bf0372 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.918018 15960 consensus_peers.cc:485] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:10.930446 13300 consensus_peers.cc:485] T a54efcd41aeb4c24adfa51bb543d9661 P 4ecb724cb4764a53988fde00407db4c0 -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.962026 16092 consensus_peers.cc:485] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:10.976747 15826 consensus_peers.cc:485] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:14:11.001268 13281 maintenance_manager.cc:419] P 6ed448d04e4646658c0de37d829ff15b: Scheduling FlushMRSOp(911cc47749484e88882c0efa75fc6396): perf score=0.033383
W20240501 08:14:11.012073 13301 consensus_peers.cc:485] T 74d1a1cd0bf94d0d9a87674d41838d2d P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:14:11.013715 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 13153
W20240501 08:14:11.020576 15960 consensus_peers.cc:485] T 03895995b2e5483998d442855006b24c P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:14:11.024811 31629 raft_consensus.cc:482] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:11.024992 31629 raft_consensus.cc:504] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:11.025609 31629 leader_election.cc:288] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:14:11.026595 13365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a412b8ae0aa648d7979fac58b50d86a6" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "4ecb724cb4764a53988fde00407db4c0" is_pre_election: true
W20240501 08:14:11.027140 15825 leader_election.cc:334] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
I20240501 08:14:11.027455 15825 leader_election.cc:302] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:11.027704 31629 raft_consensus.cc:2726] T a412b8ae0aa648d7979fac58b50d86a6 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:11.037725 15826 consensus_peers.cc:485] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:11.041937 16092 consensus_peers.cc:485] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Couldn't send request to peer e3fefd4dca2d41678187047f06c10187. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.056334 13301 consensus_peers.cc:485] T a6154bf2fadb45e58c27c46165b5824f P 4ecb724cb4764a53988fde00407db4c0 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:11.057250 16093 consensus_peers.cc:485] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:11.086313 15692 connection.cc:535] client connection to 127.6.56.67:33645 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:11.086915 13301 connection.cc:535] client connection to 127.6.56.67:33645 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:11.087033 15960 connection.cc:535] client connection to 127.6.56.67:33645 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:14:11.087342 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 13283
W20240501 08:14:11.087800 15826 connection.cc:535] client connection to 127.6.56.67:33645 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:11.088835 13301 consensus_peers.cc:485] T 74d6cce81d164722afad1935d4695374 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.088971 13301 consensus_peers.cc:485] T 04ecd52892254c99857cd1d89f514f99 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.088996 15960 consensus_peers.cc:485] T 279b5f4d8fba4c12beec040c6ed39992 P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089082 13301 consensus_peers.cc:485] T 9348009ae1444e73b6120161309dc4a2 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089107 15960 consensus_peers.cc:485] T d38e22c9e39d43bc818eede00882ff8d P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089183 13301 consensus_peers.cc:485] T 3b5240d5db124f409b28fbde8cf68f8a P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089192 15960 consensus_peers.cc:485] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089296 15960 consensus_peers.cc:485] T b0dccc451745459d8dca19f9ebe2e47f P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089285 13301 consensus_peers.cc:485] T d09d4bf8b82941199313f03bc1abeab5 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089398 13301 consensus_peers.cc:485] T 65a3d07808414c50a330da007c8f5e19 P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089398 15960 consensus_peers.cc:485] T 7c4b83a76d7b44039ebb17245fcd8a3c P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089493 13301 consensus_peers.cc:485] T f22fc6797cb24451b78a5c1e5f6306ba P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089501 15960 consensus_peers.cc:485] T 7a71b0688ca04973a46b3cbc2ce680d2 P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089576 13301 consensus_peers.cc:485] T faa5b3517ffc490cab68d09bbe2d5c0e P 4ecb724cb4764a53988fde00407db4c0 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089586 15960 consensus_peers.cc:485] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.089663 15960 consensus_peers.cc:485] T 236d2386bb2d4f378faeabce22238402 P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.090030 15692 consensus_peers.cc:485] T ba08236c71df4c8190f6b7644ed944ed P 982f49fc87f149439cda0de2261b6bfd -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.090265 16092 connection.cc:535] server connection from 127.6.56.67:42551 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:11.090521 15826 consensus_peers.cc:485] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.090646 15826 consensus_peers.cc:485] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.090760 15826 consensus_peers.cc:485] T 15dd4768e5ff47719db40ad39fbd2838 P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.090849 15826 consensus_peers.cc:485] T 80ef1222e2324309a009a657ca704795 P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.090927 15826 consensus_peers.cc:485] T 39b12a1d835d428085d4a24233d78c38 P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.091006 15826 consensus_peers.cc:485] T 37b0020baddc4f47b625edc2e1180e51 P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:11.094065 13405 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:14:11.098205 15826 consensus_peers.cc:485] T fa10a6f53f7f48cf9410c80e1056439d P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:11.109462 15826 consensus_peers.cc:485] T 8c487b2c7fe24515a8970a07883c126f P 4d516916eeac468db514eeb928f06190 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.130407 15826 consensus_peers.cc:462] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:11.148044 15692 consensus_peers.cc:485] T 949d6765e68146d998ff4f5d2b120d68 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:11.164640 15960 consensus_peers.cc:485] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:11.175657 15692 consensus_peers.cc:485] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:11.181771 15960 consensus_peers.cc:485] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:11.189664 16093 consensus_peers.cc:485] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 08:14:11.194252 15692 consensus_peers.cc:462] T 89fc6e35197e45acb3f934888784d1aa P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:11.201504 15692 consensus_peers.cc:485] T 181a5dd80e5a448b9a63d037e8ae93e7 P 982f49fc87f149439cda0de2261b6bfd -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:14:11.265437 31559 raft_consensus.cc:482] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:11.265647 31559 raft_consensus.cc:504] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:11.266290 31559 leader_election.cc:288] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
W20240501 08:14:11.267503 15959 leader_election.cc:334] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:11.268074 15960 leader_election.cc:334] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:11.268187 15960 leader_election.cc:302] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 76d830e199eb461c8ca689c9460ee0c2, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:11.268462 31559 raft_consensus.cc:2726] T db4717fa6c4e45d18702741c141f6d12 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:11.301195 31584 raft_consensus.cc:482] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:11.301396 31584 raft_consensus.cc:504] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:11.302145 31584 leader_election.cc:288] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
I20240501 08:14:11.303202 15892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99eddd74b7740dcab6b93ca437b51ae" candidate_uuid: "982f49fc87f149439cda0de2261b6bfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
W20240501 08:14:11.303839 15691 leader_election.cc:334] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
I20240501 08:14:11.303962 15691 leader_election.cc:302] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 982f49fc87f149439cda0de2261b6bfd; no voters: 4d516916eeac468db514eeb928f06190, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:11.304250 31584 raft_consensus.cc:2726] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:11.390815 15692 consensus_peers.cc:485] T 6840d246ddfa42b3b8065d7136a36f5a P 982f49fc87f149439cda0de2261b6bfd -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:11.430281 31639 raft_consensus.cc:482] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:11.430469 31639 raft_consensus.cc:504] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:11.431114 31639 leader_election.cc:288] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:11.432565 15825 leader_election.cc:334] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:11.433054 15826 leader_election.cc:334] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.433188 15826 leader_election.cc:302] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:11.433457 31639 raft_consensus.cc:2726] T 0f723c6321424530bcd41b9ecdcbe0b3 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:11.462688 15960 consensus_peers.cc:485] T ca6339e48a2f4b9e94e32e05c7bf343b P 5dd722c770d34d6c9154a87917117161 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:14:11.492147 31649 raft_consensus.cc:482] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:11.492429 31649 raft_consensus.cc:504] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:11.493203 31649 leader_election.cc:288] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4ecb724cb4764a53988fde00407db4c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:14:11.493595 15893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5fda95d9e9d48e7a4d6eb4d9b85fb57" candidate_uuid: "4ecb724cb4764a53988fde00407db4c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d516916eeac468db514eeb928f06190" is_pre_election: true
I20240501 08:14:11.495757 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 13283
I20240501 08:14:11.501770 31319 raft_consensus.cc:482] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:11.501952 31319 raft_consensus.cc:504] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:11.502671 31319 leader_election.cc:288] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:11.503954 15825 leader_election.cc:334] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:11.504328 15826 leader_election.cc:334] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.504463 15826 leader_election.cc:302] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:11.504742 31319 raft_consensus.cc:2726] T d5179549829d451eb3a6e2083ef59b7f P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:11.518885 15826 consensus_peers.cc:462] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:11.541438 31319 raft_consensus.cc:482] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:11.541639 31319 raft_consensus.cc:504] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:11.542338 31319 leader_election.cc:288] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:14:11.542953 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99eddd74b7740dcab6b93ca437b51ae" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:14:11.543181 15754 raft_consensus.cc:2454] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 1.
I20240501 08:14:11.543633 15825 leader_election.cc:302] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
W20240501 08:14:11.544366 15825 leader_election.cc:334] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
I20240501 08:14:11.544392 31319 raft_consensus.cc:2781] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 08:14:11.544508 31319 raft_consensus.cc:482] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:11.544601 31319 raft_consensus.cc:3037] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:11.547156 31319 raft_consensus.cc:504] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:11.547747 31319 leader_election.cc:288] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 election: Requested vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
I20240501 08:14:11.548612 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c99eddd74b7740dcab6b93ca437b51ae" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd"
I20240501 08:14:11.548861 15754 raft_consensus.cc:3037] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:11.552146 15754 raft_consensus.cc:2454] T c99eddd74b7740dcab6b93ca437b51ae P 982f49fc87f149439cda0de2261b6bfd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d516916eeac468db514eeb928f06190 in term 2.
I20240501 08:14:11.553282 15825 leader_election.cc:302] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d516916eeac468db514eeb928f06190, 982f49fc87f149439cda0de2261b6bfd; no voters:
W20240501 08:14:11.557204 15825 leader_election.cc:334] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
I20240501 08:14:11.557665 31319 raft_consensus.cc:2781] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:11.557874 31319 raft_consensus.cc:686] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [term 2 LEADER]: Becoming Leader. State: Replica: 4d516916eeac468db514eeb928f06190, State: Running, Role: LEADER
I20240501 08:14:11.558264 31319 consensus_queue.cc:237] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
W20240501 08:14:11.561378 15959 connection.cc:535] client connection to 127.6.56.68:36991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:11.562688 15825 connection.cc:535] client connection to 127.6.56.68:36991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:11.563820 15826 connection.cc:535] server connection from 127.6.56.68:37921 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:11.563817 16092 connection.cc:535] client connection to 127.6.56.68:36991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:14:11.564734 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 15673
W20240501 08:14:11.564886 16092 consensus_peers.cc:485] T 1eac68af4bc3473487bf91670cb48223 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.564967 31522 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:14:11.563399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.56.68:36991 (local address 127.6.56.70:36083)
0501 08:14:11.563520 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0501 08:14:11.563536 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0501 08:14:11.563646 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:14:11.564395 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":24}
W20240501 08:14:11.564992 16092 consensus_peers.cc:485] T b000ec8b190644f2883e5f824463b762 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.565057 16092 consensus_peers.cc:485] T 1ba04c9b0d0a4933a123dbeb5b50145e P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.565109 16092 consensus_peers.cc:485] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.565194 15825 consensus_peers.cc:485] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.565304 15825 consensus_peers.cc:485] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.565399 15825 consensus_peers.cc:485] T 85056837a567420a8b480b34b6017b40 P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.565508 15825 consensus_peers.cc:485] T b8a350eade1c41e4b4c29e487374552f P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.565600 15825 consensus_peers.cc:485] T 4e330f6ac7074134ba06effe5ececcde P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.565686 15825 consensus_peers.cc:485] T 66af91fdcce84b779cf6423562b582b0 P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.565774 15825 consensus_peers.cc:485] T 34a5baf3631b45d68a2003fc4539f799 P 4d516916eeac468db514eeb928f06190 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.566670 31520 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 08:14:11.562071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.56.68:36991 (local address 127.6.56.71:50027)
0501 08:14:11.562569 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0501 08:14:11.562586 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 08:14:11.562923 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 08:14:11.565072 (+ 2149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":171}
W20240501 08:14:11.566817 15959 consensus_peers.cc:485] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.566949 15959 consensus_peers.cc:485] T fc4ad07cecff43f0b867e4db444957e7 P 5dd722c770d34d6c9154a87917117161 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.567060 15959 consensus_peers.cc:485] T d241e087e754485cb80080754d27aef7 P 5dd722c770d34d6c9154a87917117161 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.567160 15959 consensus_peers.cc:485] T 4345eabfdb894def8d7063773dc5fac6 P 5dd722c770d34d6c9154a87917117161 -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.568482 15691 connection.cc:535] client connection to 127.6.56.68:36991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:11.572693 15691 consensus_peers.cc:485] T 3e62b4a817174374a1ffdf9f8b25fb44 P 982f49fc87f149439cda0de2261b6bfd -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:11.572830 15691 consensus_peers.cc:485] T 608bc35666234d2cbcb816439d050196 P 982f49fc87f149439cda0de2261b6bfd -> Peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Couldn't send request to peer 4ecb724cb4764a53988fde00407db4c0. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:11.573362 12827 catalog_manager.cc:5484] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 reported cstate change: term changed from 1 to 2, leader changed from e3fefd4dca2d41678187047f06c10187 (127.6.56.66) to 4d516916eeac468db514eeb928f06190 (127.6.56.70). New cstate: current_term: 2 leader_uuid: "4d516916eeac468db514eeb928f06190" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } }
I20240501 08:14:11.574424 15796 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:14:11.596019 31619 raft_consensus.cc:482] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.596220 31619 raft_consensus.cc:504] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:11.596920 31619 leader_election.cc:288] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
W20240501 08:14:11.601368 15959 leader_election.cc:334] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:11.601634 15960 leader_election.cc:334] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.601754 15960 leader_election.cc:302] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:11.602149 31619 raft_consensus.cc:2726] T 94e7fb526fbc4ebc94875c00b49ac43d P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:11.651296 31619 raft_consensus.cc:482] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.651496 31619 raft_consensus.cc:504] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:11.652139 31619 leader_election.cc:288] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
W20240501 08:14:11.653673 15960 leader_election.cc:334] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
W20240501 08:14:11.653868 15960 leader_election.cc:334] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:11.653957 15960 leader_election.cc:302] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:11.654208 31619 raft_consensus.cc:2726] T 89ec13cb2d2a418bba12b47f2ff4680b P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:14:11.664330 31619 raft_consensus.cc:482] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.664527 31619 raft_consensus.cc:504] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:11.665675 31619 leader_election.cc:288] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:14:11.665969 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
W20240501 08:14:11.666755 15960 leader_election.cc:334] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.666875 15960 leader_election.cc:302] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 6cc32f6e2d49477f9a23ce6e444039fb, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:11.667187 31619 raft_consensus.cc:2726] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 08:14:11.667485 15959 leader_election.cc:334] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
I20240501 08:14:11.698549 31319 raft_consensus.cc:482] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.698758 31319 raft_consensus.cc:504] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:11.699433 31319 leader_election.cc:288] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
W20240501 08:14:11.700796 15826 leader_election.cc:334] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
W20240501 08:14:11.700976 15826 leader_election.cc:334] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:11.701071 15826 leader_election.cc:302] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:11.701315 31319 raft_consensus.cc:2726] T 106da82bab514cab94c32effacf35691 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:11.713708 31319 raft_consensus.cc:482] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.713927 31319 raft_consensus.cc:504] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:11.714579 31319 leader_election.cc:288] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:11.715854 15826 leader_election.cc:334] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:11.716058 15826 leader_election.cc:334] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.716149 15826 leader_election.cc:302] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:11.716429 31319 raft_consensus.cc:2726] T a8156a46917c40e2b836a603ff70717a P 4d516916eeac468db514eeb928f06190 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:14:11.752056 31558 raft_consensus.cc:482] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.752240 31558 raft_consensus.cc:504] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:11.753048 31558 leader_election.cc:288] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:11.754740 16092 leader_election.cc:334] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:11.754856 16093 leader_election.cc:334] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.754995 16093 leader_election.cc:302] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:11.755293 31558 raft_consensus.cc:2726] T 53872527ad264dd9b4b1627ff8aa46db P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:11.755573 31558 raft_consensus.cc:482] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.755693 31558 raft_consensus.cc:504] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:11.756249 31558 leader_election.cc:288] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:14:11.756927 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
W20240501 08:14:11.757891 16093 leader_election.cc:334] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.757994 16093 leader_election.cc:302] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 5dd722c770d34d6c9154a87917117161, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:11.758174 31558 raft_consensus.cc:2726] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:14:11.863370 31619 raft_consensus.cc:482] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.863584 31619 raft_consensus.cc:504] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:11.864234 31619 leader_election.cc:288] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:14:11.864925 16158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
I20240501 08:14:11.865149 16158 raft_consensus.cc:2454] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 2.
I20240501 08:14:11.865558 15957 leader_election.cc:302] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters:
I20240501 08:14:11.865880 31559 raft_consensus.cc:2781] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 08:14:11.865985 31559 raft_consensus.cc:482] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.866098 31559 raft_consensus.cc:3037] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:14:11.867434 15960 leader_election.cc:334] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.882598 31559 raft_consensus.cc:504] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:11.883298 31559 leader_election.cc:288] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 election: Requested vote from peers 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:14:11.883651 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a713c94c6bb14d8793ee429e52f0e7a0" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb"
I20240501 08:14:11.883850 16154 raft_consensus.cc:3037] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Advancing to term 3
W20240501 08:14:11.884845 15960 leader_election.cc:334] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.886898 16154 raft_consensus.cc:2454] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd722c770d34d6c9154a87917117161 in term 3.
I20240501 08:14:11.887392 15957 leader_election.cc:302] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:11.887688 31559 raft_consensus.cc:2781] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:11.887812 31559 raft_consensus.cc:686] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [term 3 LEADER]: Becoming Leader. State: Replica: 5dd722c770d34d6c9154a87917117161, State: Running, Role: LEADER
I20240501 08:14:11.888053 31559 consensus_queue.cc:237] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:11.888835 31639 raft_consensus.cc:482] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.889039 31639 raft_consensus.cc:504] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:11.890343 31639 leader_election.cc:288] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:14:11.890774 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6fefd76544642ddb787cc6febe8df82" candidate_uuid: "4d516916eeac468db514eeb928f06190" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
W20240501 08:14:11.892259 15826 leader_election.cc:334] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.892402 15826 leader_election.cc:302] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 6cc32f6e2d49477f9a23ce6e444039fb, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:11.892678 31639 raft_consensus.cc:2726] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:11.893316 15825 leader_election.cc:334] T e6fefd76544642ddb787cc6febe8df82 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
I20240501 08:14:11.895447 12827 catalog_manager.cc:5484] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 reported cstate change: term changed from 2 to 3, leader changed from 6ed448d04e4646658c0de37d829ff15b (127.6.56.67) to 5dd722c770d34d6c9154a87917117161 (127.6.56.71). New cstate: current_term: 3 leader_uuid: "5dd722c770d34d6c9154a87917117161" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } }
I20240501 08:14:11.911315 31558 raft_consensus.cc:482] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.911494 31558 raft_consensus.cc:504] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:11.912109 31558 leader_election.cc:288] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:11.913870 16092 leader_election.cc:334] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:11.913993 16093 leader_election.cc:334] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.914101 16093 leader_election.cc:302] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:11.914299 31558 raft_consensus.cc:2726] T 5741e5458ccd4af6a3972495ccf40b8f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:11.915321 15826 consensus_peers.cc:462] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:11.921106 31558 raft_consensus.cc:482] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.921241 31558 raft_consensus.cc:504] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:11.921898 31558 leader_election.cc:288] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
W20240501 08:14:11.922969 16093 leader_election.cc:334] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.947185 15755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2794da46e543b8a25a8e65b229f95e" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "982f49fc87f149439cda0de2261b6bfd" is_pre_election: true
I20240501 08:14:11.948274 16092 leader_election.cc:302] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 6ed448d04e4646658c0de37d829ff15b, 982f49fc87f149439cda0de2261b6bfd
I20240501 08:14:11.948691 31558 raft_consensus.cc:482] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.948861 31558 raft_consensus.cc:504] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:11.949654 31558 leader_election.cc:288] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:14:11.949810 31558 raft_consensus.cc:2726] T 4e2794da46e543b8a25a8e65b229f95e P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:11.951422 16093 leader_election.cc:334] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:11.954537 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 15673
I20240501 08:14:11.955963 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aa998326f3c4ced9afbee7e54bffa0e" candidate_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5dd722c770d34d6c9154a87917117161" is_pre_election: true
I20240501 08:14:11.956234 16023 raft_consensus.cc:2454] T 6aa998326f3c4ced9afbee7e54bffa0e P 5dd722c770d34d6c9154a87917117161 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cc32f6e2d49477f9a23ce6e444039fb in term 2.
W20240501 08:14:11.957906 16092 leader_election.cc:334] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
I20240501 08:14:11.958035 16092 leader_election.cc:302] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161, 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:11.958318 31667 raft_consensus.cc:2726] T 6aa998326f3c4ced9afbee7e54bffa0e P 6cc32f6e2d49477f9a23ce6e444039fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 08:14:11.987484 31619 raft_consensus.cc:482] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:11.987691 31619 raft_consensus.cc:504] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:11.988588 31619 leader_election.cc:288] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
W20240501 08:14:11.989881 15960 leader_election.cc:334] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
W20240501 08:14:11.990585 15960 leader_election.cc:334] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:11.990712 15960 leader_election.cc:302] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 6ed448d04e4646658c0de37d829ff15b, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:11.991044 31559 raft_consensus.cc:2726] T 4fc5dbfe3a9b4c84a3217981527e73ea P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:12.001500 31559 raft_consensus.cc:482] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:12.001727 31559 raft_consensus.cc:504] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } }
I20240501 08:14:12.002254 31639 raft_consensus.cc:482] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:12.002492 31559 leader_election.cc:288] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
I20240501 08:14:12.002615 31639 raft_consensus.cc:504] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:12.003485 31639 leader_election.cc:288] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
I20240501 08:14:12.003662 31558 raft_consensus.cc:482] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:12.003825 31558 raft_consensus.cc:504] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } }
I20240501 08:14:12.004158 31667 raft_consensus.cc:482] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:12.004324 31667 raft_consensus.cc:504] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:12.004673 31558 leader_election.cc:288] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897)
W20240501 08:14:12.005048 15960 leader_election.cc:334] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:12.004915 31667 leader_election.cc:288] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
W20240501 08:14:12.005522 15959 leader_election.cc:334] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
I20240501 08:14:12.005645 15959 leader_election.cc:302] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:12.006047 31559 raft_consensus.cc:2726] T 911cc47749484e88882c0efa75fc6396 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:12.006510 15825 leader_election.cc:334] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:12.006695 15825 leader_election.cc:334] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
I20240501 08:14:12.006800 15825 leader_election.cc:302] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d516916eeac468db514eeb928f06190; no voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187
W20240501 08:14:12.007879 16092 leader_election.cc:334] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:12.008049 16092 leader_election.cc:334] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:12.008086 16093 leader_election.cc:334] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
W20240501 08:14:12.008232 16092 leader_election.cc:334] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
I20240501 08:14:12.008311 31639 raft_consensus.cc:2726] T b5fda95d9e9d48e7a4d6eb4d9b85fb57 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:12.008216 16093 leader_election.cc:302] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:12.008380 16092 leader_election.cc:302] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4ecb724cb4764a53988fde00407db4c0, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:12.008626 31558 raft_consensus.cc:2726] T 13d9aa0c21e1405791e340cd79738256 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:12.008842 31667 raft_consensus.cc:2726] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:12.010124 31667 raft_consensus.cc:482] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e3fefd4dca2d41678187047f06c10187)
I20240501 08:14:12.010272 31667 raft_consensus.cc:504] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:12.010912 31667 leader_election.cc:288] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:12.011937 16092 leader_election.cc:334] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
W20240501 08:14:12.012148 16093 leader_election.cc:334] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:12.012282 16093 leader_election.cc:302] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:12.012778 31667 raft_consensus.cc:2726] T 21719a43fa6845f8a3a5337dbe770f60 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:12.016059 16092 connection.cc:535] client connection to 127.6.56.69:42897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:12.016042 15959 connection.cc:535] client connection to 127.6.56.69:42897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:12.016620 15825 connection.cc:535] client connection to 127.6.56.69:42897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:14:12.016958 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 15805
W20240501 08:14:12.017431 15959 consensus_peers.cc:485] T 98306f9c17d8420396452ab3854d5246 P 5dd722c770d34d6c9154a87917117161 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.017480 16092 leader_election.cc:334] T 99ecb2ec0b3d4fd7a02896cc9a28ba27 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111)
W20240501 08:14:12.017573 15959 consensus_peers.cc:485] T 427bc3b61a874f83bc27d9be24732e00 P 5dd722c770d34d6c9154a87917117161 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.018687 15825 consensus_peers.cc:485] T 9102cc8e9b92469ba2ccaa25fea2cc01 P 4d516916eeac468db514eeb928f06190 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.018821 15825 consensus_peers.cc:485] T 6856858a304c4e208188897fe2ebdb33 P 4d516916eeac468db514eeb928f06190 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.018924 15825 consensus_peers.cc:485] T f8203d209d624d2695e5b32bbfa0edf7 P 4d516916eeac468db514eeb928f06190 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.019016 15825 consensus_peers.cc:485] T f305a6e24ab1436b895db037f5d72870 P 4d516916eeac468db514eeb928f06190 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.019098 15825 consensus_peers.cc:485] T 2523847da7464e9f95b18a2e88d813b4 P 4d516916eeac468db514eeb928f06190 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.020855 15825 consensus_peers.cc:485] T c99eddd74b7740dcab6b93ca437b51ae P 4d516916eeac468db514eeb928f06190 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.021378 15959 consensus_peers.cc:485] T 82e08735b5db41dcbccec46aa7278530 P 5dd722c770d34d6c9154a87917117161 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:12.024890 15933 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 08:14:12.049696 15959 consensus_peers.cc:485] T ec6d272070b140d9b918d182613008f4 P 5dd722c770d34d6c9154a87917117161 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.084190 15959 consensus_peers.cc:485] T bdabd94a46dc4a5796949fdf9d0c698c P 5dd722c770d34d6c9154a87917117161 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:12.094138 31667 raft_consensus.cc:482] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:12.094341 31667 raft_consensus.cc:504] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } }
I20240501 08:14:12.095007 31667 leader_election.cc:288] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
W20240501 08:14:12.096223 16093 leader_election.cc:334] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
W20240501 08:14:12.096668 16092 leader_election.cc:334] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
I20240501 08:14:12.096800 16092 leader_election.cc:302] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 6ed448d04e4646658c0de37d829ff15b, e3fefd4dca2d41678187047f06c10187
I20240501 08:14:12.097052 31667 raft_consensus.cc:2726] T b2b0c752d9614f2fbd3863d92df609a4 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:12.114295 16092 consensus_peers.cc:485] T cabfd31e8ee941cf9f69bc04a0510d11 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:12.122244 31667 raft_consensus.cc:482] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:12.122645 31667 raft_consensus.cc:504] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:12.123587 31667 leader_election.cc:288] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
W20240501 08:14:12.124585 16093 leader_election.cc:334] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:12.124758 16092 leader_election.cc:334] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
I20240501 08:14:12.124893 16092 leader_election.cc:302] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:12.125123 31667 raft_consensus.cc:2726] T 7a98388bf2fd402b8064d2cca06f5b8b P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:12.195853 31667 raft_consensus.cc:482] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:12.196082 31667 raft_consensus.cc:504] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } }
I20240501 08:14:12.196838 31667 leader_election.cc:288] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
W20240501 08:14:12.198172 16092 leader_election.cc:334] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
W20240501 08:14:12.198321 16093 leader_election.cc:334] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:12.198472 16093 leader_election.cc:302] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:12.198765 31667 raft_consensus.cc:2726] T 04ecd52892254c99857cd1d89f514f99 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:12.207135 15959 consensus_peers.cc:485] T 8a48ba4666c14848b2b5a1e62f16357b P 5dd722c770d34d6c9154a87917117161 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:12.364481 31667 raft_consensus.cc:482] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:12.364686 31667 raft_consensus.cc:504] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } }
I20240501 08:14:12.365384 31667 leader_election.cc:288] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:12.366813 16092 leader_election.cc:334] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
W20240501 08:14:12.366955 16093 leader_election.cc:334] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:12.367064 16093 leader_election.cc:302] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:12.367296 31667 raft_consensus.cc:2726] T 65a3d07808414c50a330da007c8f5e19 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:12.383430 31639 raft_consensus.cc:482] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:12.383648 31639 raft_consensus.cc:504] T a54efcd41aeb4c24adfa51bb543d9661 P 4d516916eeac468db514eeb928f06190 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:12.384652 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 15805
I20240501 08:14:12.381897 31667 raft_consensus.cc:482] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:12.386034 31667 raft_consensus.cc:504] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:12.386595 31667 leader_election.cc:288] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:12.387935 16093 leader_election.cc:334] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
W20240501 08:14:12.388192 16092 leader_election.cc:334] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
I20240501 08:14:12.388350 16092 leader_election.cc:302] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:12.388774 31667 raft_consensus.cc:2726] T 3b5240d5db124f409b28fbde8cf68f8a P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:12.389902 15960 consensus_peers.cc:485] T a713c94c6bb14d8793ee429e52f0e7a0 P 5dd722c770d34d6c9154a87917117161 -> Peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Couldn't send request to peer 6ed448d04e4646658c0de37d829ff15b. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:12.407506 31559 raft_consensus.cc:482] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:12.407723 31559 raft_consensus.cc:504] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:12.408425 31559 leader_election.cc:288] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6cc32f6e2d49477f9a23ce6e444039fb (127.6.56.72:45383)
I20240501 08:14:12.409050 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6658ae0ccbcc4baf8853ab8778049030" candidate_uuid: "5dd722c770d34d6c9154a87917117161" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" is_pre_election: true
W20240501 08:14:12.410241 15959 leader_election.cc:334] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
I20240501 08:14:12.410378 15959 leader_election.cc:302] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd722c770d34d6c9154a87917117161; no voters: 4ecb724cb4764a53988fde00407db4c0, 6cc32f6e2d49477f9a23ce6e444039fb
I20240501 08:14:12.410653 31559 raft_consensus.cc:2726] T 6658ae0ccbcc4baf8853ab8778049030 P 5dd722c770d34d6c9154a87917117161 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:12.414083 15959 consensus_peers.cc:485] T b2e7c8c1795d4c6d997c75e63dee6760 P 5dd722c770d34d6c9154a87917117161 -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.421360 16092 consensus_peers.cc:485] T 69d21bf5577f4b449a691ef6234376d2 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Couldn't send request to peer 982f49fc87f149439cda0de2261b6bfd. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.440430 16090 connection.cc:535] client connection to 127.6.56.70:43301 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:12.440497 15960 connection.cc:535] server connection from 127.6.56.70:43777 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 08:14:12.440668 15957 connection.cc:535] client connection to 127.6.56.70:43301 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:14:12.440936 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 15942
W20240501 08:14:12.441932 15957 consensus_peers.cc:485] T 315d149a26c14982aae10b5fb800dc82 P 5dd722c770d34d6c9154a87917117161 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.70:43301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.442071 15957 consensus_peers.cc:485] T 5e61276f6f4c40f8987747b997ac52ac P 5dd722c770d34d6c9154a87917117161 -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.70:43301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.442266 16090 consensus_peers.cc:485] T 5395a66a354241dbb5efdaa6f68f50c1 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.70:43301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 08:14:12.442385 16090 consensus_peers.cc:485] T 9259925ce64e4b93afc8b08045c19cd9 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.70:43301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:12.447199 16065 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:14:12.470944 31667 raft_consensus.cc:482] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:12.471170 31667 raft_consensus.cc:504] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:12.471808 31667 leader_election.cc:288] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:12.473184 16093 leader_election.cc:334] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
W20240501 08:14:12.473343 16092 leader_election.cc:334] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
I20240501 08:14:12.473480 16092 leader_election.cc:302] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:12.473731 31667 raft_consensus.cc:2726] T 9348009ae1444e73b6120161309dc4a2 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:12.498683 16093 consensus_peers.cc:485] T cf908880298b462391a84f5a8d4e3a6b P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Couldn't send request to peer 76d830e199eb461c8ca689c9460ee0c2. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 08:14:12.514569 31667 raft_consensus.cc:482] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:12.514775 31667 raft_consensus.cc:504] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } }
I20240501 08:14:12.515529 31667 leader_election.cc:288] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301)
W20240501 08:14:12.517397 16093 leader_election.cc:334] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
W20240501 08:14:12.517575 16090 leader_election.cc:334] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Network error: Client connection negotiation failed: client connection to 127.6.56.70:43301: connect: Connection refused (error 111)
I20240501 08:14:12.517719 16090 leader_election.cc:302] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4d516916eeac468db514eeb928f06190, 6ed448d04e4646658c0de37d829ff15b
W20240501 08:14:12.517894 16092 leader_election.cc:334] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
I20240501 08:14:12.517982 31667 raft_consensus.cc:2726] T e6fefd76544642ddb787cc6febe8df82 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:12.518821 31667 raft_consensus.cc:482] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6ed448d04e4646658c0de37d829ff15b)
I20240501 08:14:12.518963 31667 raft_consensus.cc:504] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:12.519542 31667 leader_election.cc:288] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645), 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991)
W20240501 08:14:12.521011 16092 leader_election.cc:334] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
W20240501 08:14:12.521272 16093 leader_election.cc:334] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
I20240501 08:14:12.521399 16093 leader_election.cc:302] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4ecb724cb4764a53988fde00407db4c0, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:12.521636 31667 raft_consensus.cc:2726] T 7883772f69cd4a18a2a6d568adbdeb61 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240501 08:14:12.522540 16090 consensus_peers.cc:485] T f8c3a95d53fd43b08205d0f278841896 P 6cc32f6e2d49477f9a23ce6e444039fb -> Peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Couldn't send request to peer 4d516916eeac468db514eeb928f06190. Status: Network error: Client connection negotiation failed: client connection to 127.6.56.70:43301: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 08:14:12.536263 31667 raft_consensus.cc:482] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:12.536489 31667 raft_consensus.cc:504] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:12.537165 31667 leader_election.cc:288] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
W20240501 08:14:12.538513 16093 leader_election.cc:334] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
W20240501 08:14:12.538664 16092 leader_election.cc:334] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991): Network error: Client connection negotiation failed: client connection to 127.6.56.68:36991: connect: Connection refused (error 111)
I20240501 08:14:12.538754 16092 leader_election.cc:302] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4ecb724cb4764a53988fde00407db4c0, 76d830e199eb461c8ca689c9460ee0c2
I20240501 08:14:12.539002 31667 raft_consensus.cc:2726] T a6154bf2fadb45e58c27c46165b5824f P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:12.666625 31667 raft_consensus.cc:482] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 982f49fc87f149439cda0de2261b6bfd)
I20240501 08:14:12.666841 31667 raft_consensus.cc:504] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } } peers { permanent_uuid: "982f49fc87f149439cda0de2261b6bfd" member_type: VOTER last_known_addr { host: "127.6.56.69" port: 42897 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "76d830e199eb461c8ca689c9460ee0c2" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 46839 } }
I20240501 08:14:12.667630 31667 leader_election.cc:288] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509), 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839)
W20240501 08:14:12.669117 16092 leader_election.cc:334] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 982f49fc87f149439cda0de2261b6bfd (127.6.56.69:42897): Network error: Client connection negotiation failed: client connection to 127.6.56.69:42897: connect: Connection refused (error 111)
W20240501 08:14:12.669275 16093 leader_election.cc:334] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76d830e199eb461c8ca689c9460ee0c2 (127.6.56.65:46839): Network error: Client connection negotiation failed: client connection to 127.6.56.65:46839: connect: Connection refused (error 111)
I20240501 08:14:12.669413 16093 leader_election.cc:302] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 76d830e199eb461c8ca689c9460ee0c2, 982f49fc87f149439cda0de2261b6bfd
W20240501 08:14:12.669584 16092 leader_election.cc:334] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509): Network error: Client connection negotiation failed: client connection to 127.6.56.66:35509: connect: Connection refused (error 111)
I20240501 08:14:12.669807 31667 raft_consensus.cc:2726] T 89fc6e35197e45acb3f934888784d1aa P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:12.785125 16154 raft_consensus.cc:1261] T a713c94c6bb14d8793ee429e52f0e7a0 P 6cc32f6e2d49477f9a23ce6e444039fb [term 3 FOLLOWER]: Refusing update from remote peer 5dd722c770d34d6c9154a87917117161: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 08:14:12.786299 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 15942
W20240501 08:14:12.839601 16090 connection.cc:535] server connection from 127.6.56.71:38083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 08:14:12.839891 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 16073
I20240501 08:14:12.845954 16198 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:14:13.152650 31686 raft_consensus.cc:482] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:13.152841 31686 raft_consensus.cc:504] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "5dd722c770d34d6c9154a87917117161" member_type: VOTER last_known_addr { host: "127.6.56.71" port: 36997 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:13.153522 31686 leader_election.cc:288] T 6658ae0ccbcc4baf8853ab8778049030 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), 5dd722c770d34d6c9154a87917117161 (127.6.56.71:36997)
I20240501 08:14:13.153694 31686 raft_consensus.cc:482] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4ecb724cb4764a53988fde00407db4c0)
I20240501 08:14:13.153872 31686 raft_consensus.cc:504] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4ecb724cb4764a53988fde00407db4c0" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36991 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:13.154503 31686 leader_election.cc:288] T a328773ef57444cb8147524107b15dba P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4ecb724cb4764a53988fde00407db4c0 (127.6.56.68:36991), e3fefd4dca2d41678187047f06c10187 (127.6.56.66:35509)
I20240501 08:14:13.154562 31667 raft_consensus.cc:482] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d516916eeac468db514eeb928f06190)
I20240501 08:14:13.154700 31667 raft_consensus.cc:504] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "6ed448d04e4646658c0de37d829ff15b" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 33645 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:13.155231 31667 leader_election.cc:288] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301), 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645)
W20240501 08:14:13.157411 16093 leader_election.cc:334] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6ed448d04e4646658c0de37d829ff15b (127.6.56.67:33645): Network error: Client connection negotiation failed: client connection to 127.6.56.67:33645: connect: Connection refused (error 111)
W20240501 08:14:13.160332 16090 leader_election.cc:334] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d516916eeac468db514eeb928f06190 (127.6.56.70:43301): Network error: Client connection negotiation failed: client connection to 127.6.56.70:43301: connect: Connection refused (error 111)
I20240501 08:14:13.160490 16090 leader_election.cc:302] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cc32f6e2d49477f9a23ce6e444039fb; no voters: 4d516916eeac468db514eeb928f06190, 6ed448d04e4646658c0de37d829ff15b
I20240501 08:14:13.160996 31686 raft_consensus.cc:2726] T 80ef1222e2324309a009a657ca704795 P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 08:14:13.161805 31677 raft_consensus.cc:482] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d516916eeac468db514eeb928f06190)
I20240501 08:14:13.161962 31677 raft_consensus.cc:504] T 6c5c7f2d0156405da42d25d1e18ccbfa P 6cc32f6e2d49477f9a23ce6e444039fb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d516916eeac468db514eeb928f06190" member_type: VOTER last_known_addr { host: "127.6.56.70" port: 43301 } } peers { permanent_uuid: "e3fefd4dca2d41678187047f06c10187" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 35509 } } peers { permanent_uuid: "6cc32f6e2d49477f9a23ce6e444039fb" member_type: VOTER last_known_addr { host: "127.6.56.72" port: 45383 } }
I20240501 08:14:13.162200 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 16073
I20240501 08:14:13.215608 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 12802
I20240501 08:14:13.221676 12862 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:14:13.513197 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 12802
2024-05-01T08:14:13Z chronyd exiting
I20240501 08:14:13.583290 6369 test_util.cc:175] -----------------------------------------------
I20240501 08:14:13.583380 6369 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_0.1714551039964266-6369-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/0, where GetParam() = 4-byte object <00-00 00-00> (159166 ms)
[----------] 1 test from DDLDuringRebalancingTest (159166 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/0
I20240501 08:14:13.585577 6369 test_util.cc:268] Using random seed: 309022339
I20240501 08:14:13.587355 6369 ts_itest-base.cc:112] Starting cluster with:
I20240501 08:14:13.587448 6369 ts_itest-base.cc:113] --------------
I20240501 08:14:13.587500 6369 ts_itest-base.cc:114] 8 tablet servers
I20240501 08:14:13.587567 6369 ts_itest-base.cc:115] 3 replicas per TS
I20240501 08:14:13.587620 6369 ts_itest-base.cc:116] --------------
2024-05-01T08:14:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T08:14:13Z Disabled control of system clock
I20240501 08:14:13.620172 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:36653
--webserver_interface=127.6.56.126
--webserver_port=0
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.56.126:36653
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240501 08:14:13.957273 31700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:13.957639 31700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:13.957779 31700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:13.965320 31700 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:13.965476 31700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 08:14:13.965569 31700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:13.965662 31700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 08:14:13.965736 31700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 08:14:13.974964 31700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.56.126:36653
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.56.126:36653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.56.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:13.976264 31700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:13.978016 31700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:13.989792 31705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:13.990295 31706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:13.991036 31708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:13.991808 31700 server_base.cc:1034] running on GCE node
I20240501 08:14:13.992640 31700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:13.994076 31700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:13.995270 31700 hybrid_clock.cc:648] HybridClock initialized: now 1714551253995206 us; error 52 us; skew 500 ppm
I20240501 08:14:13.995611 31700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:14.002552 31700 webserver.cc:460] Webserver started at http://127.6.56.126:34341/ using document root <none> and password file <none>
I20240501 08:14:14.003331 31700 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:14.003449 31700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:14.003816 31700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:14:14.006546 31700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a97456f304eb440dbe5e0b96256bc6fe"
format_stamp: "Formatted at 2024-05-01 08:14:13 on dist-test-slave-f1nk"
I20240501 08:14:14.007380 31700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a97456f304eb440dbe5e0b96256bc6fe"
format_stamp: "Formatted at 2024-05-01 08:14:13 on dist-test-slave-f1nk"
I20240501 08:14:14.012715 31700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:14:14.016063 31714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:14.017400 31700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:14:14.017577 31700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a97456f304eb440dbe5e0b96256bc6fe"
format_stamp: "Formatted at 2024-05-01 08:14:13 on dist-test-slave-f1nk"
I20240501 08:14:14.017783 31700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:14.052348 31700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:14.053283 31700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:14.053575 31700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:14.073035 31700 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.126:36653
I20240501 08:14:14.073052 31765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.126:36653 every 8 connection(s)
I20240501 08:14:14.074491 31700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 08:14:14.078370 31766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:14.082021 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 31700
I20240501 08:14:14.082247 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 08:14:14.088685 31766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe: Bootstrap starting.
I20240501 08:14:14.091809 31766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:14.093107 31766 log.cc:824] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe: Log is configured to *not* fsync() on all Append() calls
I20240501 08:14:14.095819 31766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe: No bootstrap required, opened a new log
I20240501 08:14:14.102442 31766 raft_consensus.cc:348] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97456f304eb440dbe5e0b96256bc6fe" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36653 } }
I20240501 08:14:14.102816 31766 raft_consensus.cc:374] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:14.102926 31766 raft_consensus.cc:729] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97456f304eb440dbe5e0b96256bc6fe, State: Initialized, Role: FOLLOWER
I20240501 08:14:14.103523 31766 consensus_queue.cc:260] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97456f304eb440dbe5e0b96256bc6fe" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36653 } }
I20240501 08:14:14.103715 31766 raft_consensus.cc:388] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 08:14:14.103847 31766 raft_consensus.cc:482] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 08:14:14.104002 31766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:14.106243 31766 raft_consensus.cc:504] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97456f304eb440dbe5e0b96256bc6fe" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36653 } }
I20240501 08:14:14.106858 31766 leader_election.cc:302] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a97456f304eb440dbe5e0b96256bc6fe; no voters:
I20240501 08:14:14.107522 31766 leader_election.cc:288] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 08:14:14.107667 31771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:14.108310 31771 raft_consensus.cc:686] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [term 1 LEADER]: Becoming Leader. State: Replica: a97456f304eb440dbe5e0b96256bc6fe, State: Running, Role: LEADER
I20240501 08:14:14.108884 31771 consensus_queue.cc:237] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97456f304eb440dbe5e0b96256bc6fe" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36653 } }
I20240501 08:14:14.109522 31766 sys_catalog.cc:564] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [sys.catalog]: configured and running, proceeding with master startup.
I20240501 08:14:14.112145 31772 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a97456f304eb440dbe5e0b96256bc6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97456f304eb440dbe5e0b96256bc6fe" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36653 } } }
I20240501 08:14:14.112145 31773 sys_catalog.cc:455] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader a97456f304eb440dbe5e0b96256bc6fe. Latest consensus state: current_term: 1 leader_uuid: "a97456f304eb440dbe5e0b96256bc6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a97456f304eb440dbe5e0b96256bc6fe" member_type: VOTER last_known_addr { host: "127.6.56.126" port: 36653 } } }
I20240501 08:14:14.112578 31773 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [sys.catalog]: This master's current role is: LEADER
I20240501 08:14:14.112581 31772 sys_catalog.cc:458] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe [sys.catalog]: This master's current role is: LEADER
I20240501 08:14:14.114440 31780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 08:14:14.119148 31780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 08:14:14.127830 31780 catalog_manager.cc:1347] Generated new cluster ID: 1781172594f14fa5a09dfad540b81d46
I20240501 08:14:14.128042 31780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 08:14:14.153088 31780 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 08:14:14.154466 31780 catalog_manager.cc:1504] Loading token signing keys...
I20240501 08:14:14.164350 31780 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a97456f304eb440dbe5e0b96256bc6fe: Generated new TSK 0
I20240501 08:14:14.165213 31780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 08:14:14.176739 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.65:0
--local_ip_for_outbound_sockets=127.6.56.65
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:14:14.519366 31790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:14.519707 31790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:14.519850 31790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:14.527583 31790 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:14.527750 31790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:14.527915 31790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.65
I20240501 08:14:14.537124 31790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.56.65
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.65
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:14.538543 31790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:14.540369 31790 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:14.553194 31795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:14.553876 31798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:14.554348 31796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:14.555202 31790 server_base.cc:1034] running on GCE node
I20240501 08:14:14.555859 31790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:14.556914 31790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:14.558142 31790 hybrid_clock.cc:648] HybridClock initialized: now 1714551254558100 us; error 40 us; skew 500 ppm
I20240501 08:14:14.558470 31790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:14.562682 31790 webserver.cc:460] Webserver started at http://127.6.56.65:33955/ using document root <none> and password file <none>
I20240501 08:14:14.563479 31790 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:14.563596 31790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:14.563932 31790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:14:14.566677 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d6949d828ece45dcb729633be6d49c97"
format_stamp: "Formatted at 2024-05-01 08:14:14 on dist-test-slave-f1nk"
I20240501 08:14:14.567473 31790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d6949d828ece45dcb729633be6d49c97"
format_stamp: "Formatted at 2024-05-01 08:14:14 on dist-test-slave-f1nk"
I20240501 08:14:14.572777 31790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20240501 08:14:14.576161 31804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:14.577529 31790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:14:14.577693 31790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d6949d828ece45dcb729633be6d49c97"
format_stamp: "Formatted at 2024-05-01 08:14:14 on dist-test-slave-f1nk"
I20240501 08:14:14.577896 31790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:14.600982 31790 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:14.601908 31790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:14.602183 31790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:14.603125 31790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:14.604671 31790 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:14.604772 31790 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:14.604889 31790 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:14.604962 31790 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:14.636875 31790 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.65:38515
I20240501 08:14:14.636927 31916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.65:38515 every 8 connection(s)
I20240501 08:14:14.638233 31790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 08:14:14.642673 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 31790
I20240501 08:14:14.642866 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 08:14:14.645962 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.66:0
--local_ip_for_outbound_sockets=127.6.56.66
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:14.651029 31917 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:14.651368 31917 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:14.652307 31917 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:14.654323 31731 ts_manager.cc:194] Registered new tserver with Master: d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:14.655913 31731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.65:55571
W20240501 08:14:14.982023 31921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:14.982363 31921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:14.982512 31921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:14.990322 31921 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:14.990489 31921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:14.990664 31921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.66
I20240501 08:14:14.999934 31921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.56.66
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.66
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:15.001351 31921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:15.003119 31921 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:15.016158 31927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:15.016613 31926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:15.017340 31929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:15.018031 31921 server_base.cc:1034] running on GCE node
I20240501 08:14:15.018675 31921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:15.019652 31921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:15.020872 31921 hybrid_clock.cc:648] HybridClock initialized: now 1714551255020821 us; error 48 us; skew 500 ppm
I20240501 08:14:15.021193 31921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:15.023690 31921 webserver.cc:460] Webserver started at http://127.6.56.66:33155/ using document root <none> and password file <none>
I20240501 08:14:15.024504 31921 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:15.024631 31921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:15.024966 31921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:14:15.030463 31921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6b7faa87de354980a76c80fb864f9edf"
format_stamp: "Formatted at 2024-05-01 08:14:15 on dist-test-slave-f1nk"
I20240501 08:14:15.031271 31921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6b7faa87de354980a76c80fb864f9edf"
format_stamp: "Formatted at 2024-05-01 08:14:15 on dist-test-slave-f1nk"
I20240501 08:14:15.036746 31921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 08:14:15.040258 31935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:15.041687 31921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:14:15.041857 31921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6b7faa87de354980a76c80fb864f9edf"
format_stamp: "Formatted at 2024-05-01 08:14:15 on dist-test-slave-f1nk"
I20240501 08:14:15.042060 31921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:15.062288 31921 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:15.063189 31921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:15.063467 31921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:15.064460 31921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:15.066105 31921 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:15.066220 31921 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:15.066339 31921 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:15.066417 31921 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:15.100713 31921 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.66:34121
I20240501 08:14:15.100762 32047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.66:34121 every 8 connection(s)
I20240501 08:14:15.102077 31921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 08:14:15.110816 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 31921
I20240501 08:14:15.111042 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 08:14:15.114686 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.67:0
--local_ip_for_outbound_sockets=127.6.56.67
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:15.115758 32048 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:15.116066 32048 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:15.116858 32048 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:15.118345 31731 ts_manager.cc:194] Registered new tserver with Master: 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:15.119477 31731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.66:33903
W20240501 08:14:15.467031 32051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:15.467386 32051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:15.467523 32051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:15.475242 32051 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:15.475423 32051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:15.475600 32051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.67
I20240501 08:14:15.484906 32051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.56.67
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.67
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:15.486356 32051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:15.488263 32051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:15.501142 32056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:15.501782 32057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:15.502292 32059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:15.503161 32051 server_base.cc:1034] running on GCE node
I20240501 08:14:15.503813 32051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:15.504878 32051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:15.506104 32051 hybrid_clock.cc:648] HybridClock initialized: now 1714551255506053 us; error 49 us; skew 500 ppm
I20240501 08:14:15.506453 32051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:15.508949 32051 webserver.cc:460] Webserver started at http://127.6.56.67:45423/ using document root <none> and password file <none>
I20240501 08:14:15.509717 32051 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:15.509853 32051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:15.510182 32051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:14:15.512940 32051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4827555155084237b0199283e5896782"
format_stamp: "Formatted at 2024-05-01 08:14:15 on dist-test-slave-f1nk"
I20240501 08:14:15.513746 32051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4827555155084237b0199283e5896782"
format_stamp: "Formatted at 2024-05-01 08:14:15 on dist-test-slave-f1nk"
I20240501 08:14:15.519555 32051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:14:15.523509 32065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:15.525151 32051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:14:15.525332 32051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4827555155084237b0199283e5896782"
format_stamp: "Formatted at 2024-05-01 08:14:15 on dist-test-slave-f1nk"
I20240501 08:14:15.525547 32051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:15.549973 32051 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:15.550860 32051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:15.551147 32051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:15.552129 32051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:15.553725 32051 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:15.553833 32051 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:15.553958 32051 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:15.554031 32051 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:15.585693 32051 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.67:41473
I20240501 08:14:15.585770 32177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.67:41473 every 8 connection(s)
I20240501 08:14:15.587249 32051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 08:14:15.589064 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 32051
I20240501 08:14:15.589259 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 08:14:15.592480 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.68:0
--local_ip_for_outbound_sockets=127.6.56.68
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:15.603029 32178 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:15.603322 32178 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:15.604080 32178 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:15.605458 31731 ts_manager.cc:194] Registered new tserver with Master: 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:15.606245 31731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.67:39877
I20240501 08:14:15.659447 31917 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
W20240501 08:14:15.933020 32181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:15.933337 32181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:15.933483 32181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:15.941076 32181 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:15.941236 32181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:15.941403 32181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.68
I20240501 08:14:15.950757 32181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.56.68
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.68
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:15.952165 32181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:15.954052 32181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:15.966653 32186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:15.967226 32187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:15.968142 32189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:15.968807 32181 server_base.cc:1034] running on GCE node
I20240501 08:14:15.969408 32181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:15.970429 32181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:15.971632 32181 hybrid_clock.cc:648] HybridClock initialized: now 1714551255971573 us; error 54 us; skew 500 ppm
I20240501 08:14:15.971964 32181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:15.974287 32181 webserver.cc:460] Webserver started at http://127.6.56.68:46051/ using document root <none> and password file <none>
I20240501 08:14:15.975033 32181 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:15.975142 32181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:15.975463 32181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:14:15.978159 32181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "64f7786244bd4f90af1ad476966482c4"
format_stamp: "Formatted at 2024-05-01 08:14:15 on dist-test-slave-f1nk"
I20240501 08:14:15.978933 32181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "64f7786244bd4f90af1ad476966482c4"
format_stamp: "Formatted at 2024-05-01 08:14:15 on dist-test-slave-f1nk"
I20240501 08:14:15.984601 32181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 08:14:15.988490 32195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:15.989951 32181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:14:15.990123 32181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "64f7786244bd4f90af1ad476966482c4"
format_stamp: "Formatted at 2024-05-01 08:14:15 on dist-test-slave-f1nk"
I20240501 08:14:15.990340 32181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:16.025390 32181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:16.026311 32181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:16.026588 32181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:16.027601 32181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:16.029161 32181 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:16.029256 32181 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:16.029363 32181 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:16.029431 32181 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:16.061272 32181 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.68:36729
I20240501 08:14:16.061348 32307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.68:36729 every 8 connection(s)
I20240501 08:14:16.062635 32181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 08:14:16.068032 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 32181
I20240501 08:14:16.068238 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 08:14:16.071966 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.69:0
--local_ip_for_outbound_sockets=127.6.56.69
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:16.077287 32308 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:16.077634 32308 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:16.078421 32308 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:16.079849 31731 ts_manager.cc:194] Registered new tserver with Master: 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:16.080991 31731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.68:42505
I20240501 08:14:16.122042 32048 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
W20240501 08:14:16.416558 32311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:16.416867 32311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:16.417012 32311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:16.424767 32311 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:16.424913 32311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:16.425091 32311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.69
I20240501 08:14:16.434960 32311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.56.69
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.69
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:16.436398 32311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:16.438192 32311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:16.450855 32316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:16.451391 32317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:16.452013 32319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:16.452823 32311 server_base.cc:1034] running on GCE node
I20240501 08:14:16.453475 32311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:16.454520 32311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:16.455739 32311 hybrid_clock.cc:648] HybridClock initialized: now 1714551256455689 us; error 53 us; skew 500 ppm
I20240501 08:14:16.456065 32311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:16.458703 32311 webserver.cc:460] Webserver started at http://127.6.56.69:44197/ using document root <none> and password file <none>
I20240501 08:14:16.459460 32311 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:16.459581 32311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:16.459914 32311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:14:16.462656 32311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b9c992291081480abb54b0415c9385f0"
format_stamp: "Formatted at 2024-05-01 08:14:16 on dist-test-slave-f1nk"
I20240501 08:14:16.463464 32311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b9c992291081480abb54b0415c9385f0"
format_stamp: "Formatted at 2024-05-01 08:14:16 on dist-test-slave-f1nk"
I20240501 08:14:16.468962 32311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20240501 08:14:16.472532 32325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:16.473937 32311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 08:14:16.474107 32311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b9c992291081480abb54b0415c9385f0"
format_stamp: "Formatted at 2024-05-01 08:14:16 on dist-test-slave-f1nk"
I20240501 08:14:16.474320 32311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:16.497627 32311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:16.498510 32311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:16.498790 32311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:16.499748 32311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:16.501307 32311 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:16.501418 32311 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:16.501533 32311 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:16.501613 32311 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:16.533546 32311 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.69:34467
I20240501 08:14:16.533634 32437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.69:34467 every 8 connection(s)
I20240501 08:14:16.534961 32311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:14:16.538991 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 32311
I20240501 08:14:16.539171 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 08:14:16.543100 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.70:0
--local_ip_for_outbound_sockets=127.6.56.70
--webserver_interface=127.6.56.70
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:16.549911 32438 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:16.550240 32438 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:16.551023 32438 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:16.552493 31731 ts_manager.cc:194] Registered new tserver with Master: b9c992291081480abb54b0415c9385f0 (127.6.56.69:34467)
I20240501 08:14:16.553259 31731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.69:54767
I20240501 08:14:16.608749 32178 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
W20240501 08:14:16.890816 32441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:16.891122 32441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:16.891242 32441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:16.898906 32441 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:16.899039 32441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:16.899236 32441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.70
I20240501 08:14:16.908447 32441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.56.70
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.70
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:16.909858 32441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:16.911717 32441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:16.924391 32446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:16.924855 32447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:16.925485 32449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:16.926272 32441 server_base.cc:1034] running on GCE node
I20240501 08:14:16.926883 32441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:16.927853 32441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:16.929062 32441 hybrid_clock.cc:648] HybridClock initialized: now 1714551256929012 us; error 53 us; skew 500 ppm
I20240501 08:14:16.929370 32441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:16.931655 32441 webserver.cc:460] Webserver started at http://127.6.56.70:38207/ using document root <none> and password file <none>
I20240501 08:14:16.932444 32441 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:16.932571 32441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:16.932902 32441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:14:16.935425 32441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "a86d83e684ff407fac40b08fa600a85b"
format_stamp: "Formatted at 2024-05-01 08:14:16 on dist-test-slave-f1nk"
I20240501 08:14:16.936203 32441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "a86d83e684ff407fac40b08fa600a85b"
format_stamp: "Formatted at 2024-05-01 08:14:16 on dist-test-slave-f1nk"
I20240501 08:14:16.941337 32441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240501 08:14:16.944794 32455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:16.946125 32441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20240501 08:14:16.946285 32441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a86d83e684ff407fac40b08fa600a85b"
format_stamp: "Formatted at 2024-05-01 08:14:16 on dist-test-slave-f1nk"
I20240501 08:14:16.946496 32441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:16.969323 32441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:16.970199 32441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:16.970465 32441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:16.971413 32441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:16.973001 32441 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:16.973102 32441 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:16.973222 32441 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:16.973299 32441 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:17.006373 32441 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.70:34611
I20240501 08:14:17.006459 32567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.70:34611 every 8 connection(s)
I20240501 08:14:17.007797 32441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:14:17.017877 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 32441
I20240501 08:14:17.018108 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 08:14:17.020632 32568 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:17.020947 32568 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:17.021364 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.71:0
--local_ip_for_outbound_sockets=127.6.56.71
--webserver_interface=127.6.56.71
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:17.021771 32568 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:17.023192 31731 ts_manager.cc:194] Registered new tserver with Master: a86d83e684ff407fac40b08fa600a85b (127.6.56.70:34611)
I20240501 08:14:17.024019 31731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.70:40429
I20240501 08:14:17.084511 32308 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
W20240501 08:14:17.367671 32571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:17.367988 32571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:17.368088 32571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:17.375779 32571 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:17.375926 32571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:17.376044 32571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.71
I20240501 08:14:17.385749 32571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.56.71
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.71
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:17.387092 32571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:17.388891 32571 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:17.401468 32576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:17.401942 32579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:17.402690 32577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:17.403528 32571 server_base.cc:1034] running on GCE node
I20240501 08:14:17.404170 32571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:17.405205 32571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:17.406410 32571 hybrid_clock.cc:648] HybridClock initialized: now 1714551257406358 us; error 45 us; skew 500 ppm
I20240501 08:14:17.406751 32571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:17.409319 32571 webserver.cc:460] Webserver started at http://127.6.56.71:45545/ using document root <none> and password file <none>
I20240501 08:14:17.410125 32571 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:17.410239 32571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:17.410583 32571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:14:17.413293 32571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f065d94513dc485293fd2e673e08918a"
format_stamp: "Formatted at 2024-05-01 08:14:17 on dist-test-slave-f1nk"
I20240501 08:14:17.414086 32571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f065d94513dc485293fd2e673e08918a"
format_stamp: "Formatted at 2024-05-01 08:14:17 on dist-test-slave-f1nk"
I20240501 08:14:17.419299 32571 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 08:14:17.422772 32585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:17.424131 32571 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 08:14:17.424332 32571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f065d94513dc485293fd2e673e08918a"
format_stamp: "Formatted at 2024-05-01 08:14:17 on dist-test-slave-f1nk"
I20240501 08:14:17.424557 32571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:17.440725 32571 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:17.441632 32571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:17.441918 32571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:17.442890 32571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:17.444617 32571 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:17.444737 32571 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:17.444850 32571 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:17.444928 32571 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:17.477736 32571 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.71:44621
I20240501 08:14:17.477785 32697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.71:44621 every 8 connection(s)
I20240501 08:14:17.479099 32571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:14:17.484066 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 32571
I20240501 08:14:17.484231 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 08:14:17.487329 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.72:0
--local_ip_for_outbound_sockets=127.6.56.72
--webserver_interface=127.6.56.72
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:17.492539 32698 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:17.492871 32698 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:17.493743 32698 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:17.495283 31731 ts_manager.cc:194] Registered new tserver with Master: f065d94513dc485293fd2e673e08918a (127.6.56.71:44621)
I20240501 08:14:17.496095 31731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.71:46157
I20240501 08:14:17.555878 32438 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
W20240501 08:14:17.830888 32701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:17.831240 32701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:17.831394 32701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:17.838979 32701 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:17.839154 32701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:17.839321 32701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.72
I20240501 08:14:17.849062 32701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.56.72
--webserver_port=0
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.72
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:17.850538 32701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:17.852545 32701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:17.865389 32707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:17.865744 32706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:17.866387 32709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:17.867038 32701 server_base.cc:1034] running on GCE node
I20240501 08:14:17.867720 32701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:17.868777 32701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:17.869968 32701 hybrid_clock.cc:648] HybridClock initialized: now 1714551257869923 us; error 49 us; skew 500 ppm
I20240501 08:14:17.870290 32701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:17.872713 32701 webserver.cc:460] Webserver started at http://127.6.56.72:45979/ using document root <none> and password file <none>
I20240501 08:14:17.873497 32701 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:17.873608 32701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:17.873961 32701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 08:14:17.876750 32701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e4e49c68247b499297c4fbeaf145e11e"
format_stamp: "Formatted at 2024-05-01 08:14:17 on dist-test-slave-f1nk"
I20240501 08:14:17.877542 32701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e4e49c68247b499297c4fbeaf145e11e"
format_stamp: "Formatted at 2024-05-01 08:14:17 on dist-test-slave-f1nk"
I20240501 08:14:17.883049 32701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 08:14:17.886595 32715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:17.888058 32701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:17.888231 32701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e4e49c68247b499297c4fbeaf145e11e"
format_stamp: "Formatted at 2024-05-01 08:14:17 on dist-test-slave-f1nk"
I20240501 08:14:17.888504 32701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:17.908675 32701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:17.909651 32701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:17.909947 32701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:17.911312 32701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:17.913069 32701 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:17.913169 32701 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:17.913290 32701 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:17.913362 32701 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:17.945627 32701 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.72:42943
I20240501 08:14:17.945679 359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.72:42943 every 8 connection(s)
I20240501 08:14:17.947046 32701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:14:17.951992 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 32701
I20240501 08:14:17.952172 6369 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 08:14:17.960546 360 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:17.960855 360 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:17.961591 360 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:17.962958 31731 ts_manager.cc:194] Registered new tserver with Master: e4e49c68247b499297c4fbeaf145e11e (127.6.56.72:42943)
I20240501 08:14:17.963721 31731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.72:59253
I20240501 08:14:17.968120 6369 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:14:17.985545 6369 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 08:14:17.985689 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 32311
I20240501 08:14:17.994930 32433 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:14:18.026647 32568 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
I20240501 08:14:18.078425 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 32311
I20240501 08:14:18.097530 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 32441
I20240501 08:14:18.106035 32563 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:14:18.187578 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 32441
I20240501 08:14:18.204511 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 32571
I20240501 08:14:18.212831 32693 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:14:18.293609 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 32571
I20240501 08:14:18.309911 6369 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu pid 32701
I20240501 08:14:18.318733 355 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 08:14:18.401326 6369 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu with pid 32701
I20240501 08:14:22.169751 31731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:22.220311 31850 tablet_service.cc:1467] Processing CreateTablet for tablet d92472b3840640e99f9af90c05ae7cee (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a16ec0364644230b5bff004b11535ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:22.220724 32242 tablet_service.cc:1467] Processing CreateTablet for tablet d92472b3840640e99f9af90c05ae7cee (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a16ec0364644230b5bff004b11535ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:22.222347 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d92472b3840640e99f9af90c05ae7cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.222795 32243 tablet_service.cc:1467] Processing CreateTablet for tablet f075dd8fe7f74d108cd84f578643e94d (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a16ec0364644230b5bff004b11535ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:22.221575 31852 tablet_service.cc:1467] Processing CreateTablet for tablet f075dd8fe7f74d108cd84f578643e94d (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a16ec0364644230b5bff004b11535ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:22.221992 31851 tablet_service.cc:1467] Processing CreateTablet for tablet ae6229e15d2b4adf8f25df4078b2f533 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a16ec0364644230b5bff004b11535ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:22.223860 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f075dd8fe7f74d108cd84f578643e94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.224262 31850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d92472b3840640e99f9af90c05ae7cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.224992 31851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6229e15d2b4adf8f25df4078b2f533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.229318 32112 tablet_service.cc:1467] Processing CreateTablet for tablet ae6229e15d2b4adf8f25df4078b2f533 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a16ec0364644230b5bff004b11535ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:22.229903 32113 tablet_service.cc:1467] Processing CreateTablet for tablet f075dd8fe7f74d108cd84f578643e94d (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a16ec0364644230b5bff004b11535ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:22.230163 31982 tablet_service.cc:1467] Processing CreateTablet for tablet d92472b3840640e99f9af90c05ae7cee (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a16ec0364644230b5bff004b11535ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:22.230881 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6229e15d2b4adf8f25df4078b2f533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.231391 31982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d92472b3840640e99f9af90c05ae7cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.232069 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f075dd8fe7f74d108cd84f578643e94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.232975 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f075dd8fe7f74d108cd84f578643e94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.229866 31983 tablet_service.cc:1467] Processing CreateTablet for tablet ae6229e15d2b4adf8f25df4078b2f533 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7a16ec0364644230b5bff004b11535ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:22.236860 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae6229e15d2b4adf8f25df4078b2f533. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:22.248193 389 tablet_bootstrap.cc:492] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:22.252725 389 tablet_bootstrap.cc:654] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:22.254294 390 tablet_bootstrap.cc:492] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:22.254315 389 log.cc:824] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4: Log is configured to *not* fsync() on all Append() calls
I20240501 08:14:22.259500 390 tablet_bootstrap.cc:654] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:22.260927 391 tablet_bootstrap.cc:492] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:22.261044 390 log.cc:824] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Log is configured to *not* fsync() on all Append() calls
I20240501 08:14:22.263168 393 tablet_bootstrap.cc:492] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:22.264526 389 tablet_bootstrap.cc:492] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:22.264653 391 tablet_bootstrap.cc:654] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:22.264958 389 ts_tablet_manager.cc:1367] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20240501 08:14:22.266697 391 log.cc:824] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782: Log is configured to *not* fsync() on all Append() calls
I20240501 08:14:22.267726 393 tablet_bootstrap.cc:654] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:22.269835 393 log.cc:824] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97: Log is configured to *not* fsync() on all Append() calls
I20240501 08:14:22.270484 390 tablet_bootstrap.cc:492] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:22.270598 391 tablet_bootstrap.cc:492] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:22.270817 390 ts_tablet_manager.cc:1367] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.000s
I20240501 08:14:22.270960 391 ts_tablet_manager.cc:1367] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 08:14:22.274922 389 raft_consensus.cc:348] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:22.275547 389 raft_consensus.cc:374] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:22.275696 389 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.276631 389 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:22.282269 389 ts_tablet_manager.cc:1398] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20240501 08:14:22.283304 393 tablet_bootstrap.cc:492] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:22.283630 393 ts_tablet_manager.cc:1367] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.004s
I20240501 08:14:22.283982 391 raft_consensus.cc:348] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:22.283980 390 raft_consensus.cc:348] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:22.284476 391 raft_consensus.cc:374] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:22.284487 390 raft_consensus.cc:374] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:22.284576 391 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.284580 390 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.285384 391 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:22.285392 390 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:22.290673 389 tablet_bootstrap.cc:492] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:22.293449 389 tablet_bootstrap.cc:654] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:22.293934 391 ts_tablet_manager.cc:1398] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.023s user 0.017s sys 0.004s
I20240501 08:14:22.294446 391 tablet_bootstrap.cc:492] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:22.294709 393 raft_consensus.cc:348] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:22.295354 393 raft_consensus.cc:374] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:22.295519 393 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.296339 391 tablet_bootstrap.cc:654] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:22.296535 393 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:22.299538 389 tablet_bootstrap.cc:492] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:22.299755 389 ts_tablet_manager.cc:1367] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240501 08:14:22.300550 390 ts_tablet_manager.cc:1398] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.030s user 0.011s sys 0.008s
I20240501 08:14:22.301046 390 tablet_bootstrap.cc:492] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:22.300966 389 raft_consensus.cc:348] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:22.301293 389 raft_consensus.cc:374] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:22.301425 389 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.301744 389 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:22.302927 390 tablet_bootstrap.cc:654] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:22.303438 389 ts_tablet_manager.cc:1398] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:14:22.305048 390 tablet_bootstrap.cc:492] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:22.305249 390 ts_tablet_manager.cc:1367] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 08:14:22.306279 390 raft_consensus.cc:348] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:22.306546 390 raft_consensus.cc:374] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:22.306708 390 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.306991 390 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:22.307608 390 ts_tablet_manager.cc:1398] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:22.309851 397 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:22.310083 397 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:22.311945 397 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:22.315544 393 ts_tablet_manager.cc:1398] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.032s user 0.018s sys 0.003s
I20240501 08:14:22.316110 393 tablet_bootstrap.cc:492] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:22.317237 391 tablet_bootstrap.cc:492] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
W20240501 08:14:22.318717 32309 tablet.cc:2369] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:14:22.318921 391 ts_tablet_manager.cc:1367] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.008s
I20240501 08:14:22.320544 393 tablet_bootstrap.cc:654] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:22.320531 391 raft_consensus.cc:348] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:22.320804 391 raft_consensus.cc:374] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:22.320942 391 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.321239 391 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:22.322183 391 ts_tablet_manager.cc:1398] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:22.324723 393 tablet_bootstrap.cc:492] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:22.325654 393 ts_tablet_manager.cc:1367] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20240501 08:14:22.326551 393 raft_consensus.cc:348] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:22.326809 393 raft_consensus.cc:374] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:22.326964 393 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.327292 393 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:22.327994 393 ts_tablet_manager.cc:1398] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:22.328265 393 tablet_bootstrap.cc:492] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:22.330102 393 tablet_bootstrap.cc:654] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:22.334877 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:22.335505 32133 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:22.336203 32198 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:22.336663 397 raft_consensus.cc:2781] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:22.336791 397 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:22.336620 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:22.336895 397 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:22.337057 393 tablet_bootstrap.cc:492] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:22.337235 393 ts_tablet_manager.cc:1367] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 08:14:22.337227 31872 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:22.338052 393 raft_consensus.cc:348] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:22.338250 393 raft_consensus.cc:374] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:22.338326 393 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:22.338564 393 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:22.339076 393 ts_tablet_manager.cc:1398] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:14:22.339432 397 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:22.340162 397 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:22.340837 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:22.340957 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:22.341032 31872 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:22.341157 32133 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:22.343147 31872 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:22.343199 32133 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:22.343643 32198 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:22.343976 397 raft_consensus.cc:2781] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election won for term 1
W20240501 08:14:22.344436 32179 tablet.cc:2369] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:14:22.344539 397 raft_consensus.cc:686] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:22.345126 397 consensus_queue.cc:237] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:22.346442 401 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:22.346606 401 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:22.348371 401 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:22.351318 31729 catalog_manager.cc:5484] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:22.355314 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:22.355593 32133 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:22.356240 31807 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:22.356735 401 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:22.356873 401 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:22.356969 401 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:22.358624 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:22.359189 32003 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
W20240501 08:14:22.359581 32049 tablet.cc:2369] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:14:22.359737 401 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:22.360421 401 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:22.360916 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:22.361084 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:22.361156 32133 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:22.361275 32003 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:22.363998 32133 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:22.364210 32003 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:22.364482 31807 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:22.364777 401 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:22.365209 401 raft_consensus.cc:686] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:22.365769 401 consensus_queue.cc:237] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:22.370311 31729 catalog_manager.cc:5484] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
W20240501 08:14:22.396226 31918 tablet.cc:2369] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 08:14:22.533365 397 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:22.533634 397 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:22.534356 397 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:22.534996 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:22.535277 31872 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:22.535733 32199 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:22.536007 397 raft_consensus.cc:2781] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:22.536094 397 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:22.536201 397 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:22.538797 397 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:22.539461 397 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:22.539913 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:22.540117 31872 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:22.540966 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:22.541102 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:22.541245 32003 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:22.541370 32002 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:22.542358 31872 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:22.542716 32199 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:22.543048 397 raft_consensus.cc:2781] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:22.543226 397 raft_consensus.cc:686] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:22.543538 397 consensus_queue.cc:237] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:22.543921 32002 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:22.546413 31729 catalog_manager.cc:5484] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:22.777245 397 consensus_queue.cc:1035] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:22.784265 397 consensus_queue.cc:1035] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:22.799821 401 consensus_queue.cc:1035] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:22.806221 401 consensus_queue.cc:1035] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:22.914261 415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:22.914618 415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:22.922288 415 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 08:14:23.017956 397 consensus_queue.cc:1035] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:23.035815 408 consensus_queue.cc:1035] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 275.404 ms
time per row: 0.137702 ms
I20240501 08:14:23.347616 31725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:23.356012 32243 tablet_service.cc:1467] Processing CreateTablet for tablet d816475cd23d4cf1967ef9a5f6a1ac18 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a89c97212ca6478eb5630a0bce3fb4ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:23.355989 31982 tablet_service.cc:1467] Processing CreateTablet for tablet d816475cd23d4cf1967ef9a5f6a1ac18 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a89c97212ca6478eb5630a0bce3fb4ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:23.356463 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d816475cd23d4cf1967ef9a5f6a1ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.356477 31982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d816475cd23d4cf1967ef9a5f6a1ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.357336 31851 tablet_service.cc:1467] Processing CreateTablet for tablet 93a196489f264058a064dc84690ba947 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a89c97212ca6478eb5630a0bce3fb4ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:23.357756 31851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93a196489f264058a064dc84690ba947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.359180 31983 tablet_service.cc:1467] Processing CreateTablet for tablet a9de4d2812b344acb197768825bd4f04 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a89c97212ca6478eb5630a0bce3fb4ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:23.359200 32113 tablet_service.cc:1467] Processing CreateTablet for tablet a9de4d2812b344acb197768825bd4f04 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a89c97212ca6478eb5630a0bce3fb4ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:23.359537 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9de4d2812b344acb197768825bd4f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.359586 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9de4d2812b344acb197768825bd4f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.360118 31981 tablet_service.cc:1467] Processing CreateTablet for tablet 93a196489f264058a064dc84690ba947 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a89c97212ca6478eb5630a0bce3fb4ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:23.360581 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93a196489f264058a064dc84690ba947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.360754 32111 tablet_service.cc:1467] Processing CreateTablet for tablet 93a196489f264058a064dc84690ba947 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a89c97212ca6478eb5630a0bce3fb4ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:23.361137 32111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93a196489f264058a064dc84690ba947. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.364715 32112 tablet_service.cc:1467] Processing CreateTablet for tablet d816475cd23d4cf1967ef9a5f6a1ac18 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a89c97212ca6478eb5630a0bce3fb4ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:23.365087 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d816475cd23d4cf1967ef9a5f6a1ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.368956 32242 tablet_service.cc:1467] Processing CreateTablet for tablet a9de4d2812b344acb197768825bd4f04 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a89c97212ca6478eb5630a0bce3fb4ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:23.369390 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9de4d2812b344acb197768825bd4f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:23.370774 475 tablet_bootstrap.cc:492] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:23.373440 475 tablet_bootstrap.cc:654] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:23.378068 478 tablet_bootstrap.cc:492] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:23.380153 479 tablet_bootstrap.cc:492] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:23.381516 478 tablet_bootstrap.cc:654] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:23.385082 479 tablet_bootstrap.cc:654] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:23.390295 476 tablet_bootstrap.cc:492] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:23.391726 479 tablet_bootstrap.cc:492] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:23.391853 475 tablet_bootstrap.cc:492] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:23.392055 475 ts_tablet_manager.cc:1367] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.008s
I20240501 08:14:23.392324 479 ts_tablet_manager.cc:1367] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240501 08:14:23.392438 478 tablet_bootstrap.cc:492] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:23.392639 478 ts_tablet_manager.cc:1367] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20240501 08:14:23.393000 475 raft_consensus.cc:348] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.393235 475 raft_consensus.cc:374] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:23.393177 479 raft_consensus.cc:348] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:23.393357 475 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.393401 479 raft_consensus.cc:374] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:23.393288 478 raft_consensus.cc:348] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.393491 478 raft_consensus.cc:374] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:23.393500 479 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.393590 478 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.393671 475 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.393721 479 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:23.394584 479 ts_tablet_manager.cc:1398] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:23.394922 479 tablet_bootstrap.cc:492] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:23.395087 478 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.395483 476 tablet_bootstrap.cc:654] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:23.395532 475 ts_tablet_manager.cc:1398] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:14:23.395860 475 tablet_bootstrap.cc:492] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:23.397192 478 ts_tablet_manager.cc:1398] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:14:23.398187 475 tablet_bootstrap.cc:654] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:23.401005 476 tablet_bootstrap.cc:492] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:23.401232 476 ts_tablet_manager.cc:1367] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240501 08:14:23.401775 479 tablet_bootstrap.cc:654] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:23.402202 476 raft_consensus.cc:348] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:23.402472 476 raft_consensus.cc:374] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:23.402621 476 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.402997 476 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:23.403764 476 ts_tablet_manager.cc:1398] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:23.404117 476 tablet_bootstrap.cc:492] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:23.405870 476 tablet_bootstrap.cc:654] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:23.406935 475 tablet_bootstrap.cc:492] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:23.407261 475 ts_tablet_manager.cc:1367] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240501 08:14:23.408440 475 raft_consensus.cc:348] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:23.408843 475 raft_consensus.cc:374] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:23.408998 475 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.409449 475 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:23.411108 475 ts_tablet_manager.cc:1398] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:14:23.412699 479 tablet_bootstrap.cc:492] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:23.413002 479 ts_tablet_manager.cc:1367] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.000s
I20240501 08:14:23.413882 479 raft_consensus.cc:348] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.414213 479 raft_consensus.cc:374] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:23.414304 479 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.414505 476 tablet_bootstrap.cc:492] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:23.414693 476 ts_tablet_manager.cc:1367] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240501 08:14:23.414621 479 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.415271 479 ts_tablet_manager.cc:1398] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:23.415483 476 raft_consensus.cc:348] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.415637 479 tablet_bootstrap.cc:492] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:23.415724 476 raft_consensus.cc:374] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:23.415807 476 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.416054 476 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.416632 476 ts_tablet_manager.cc:1398] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:23.416960 476 tablet_bootstrap.cc:492] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:23.417398 479 tablet_bootstrap.cc:654] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:23.418512 476 tablet_bootstrap.cc:654] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:23.421991 479 tablet_bootstrap.cc:492] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:23.421991 476 tablet_bootstrap.cc:492] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:23.422140 479 ts_tablet_manager.cc:1367] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240501 08:14:23.422140 476 ts_tablet_manager.cc:1367] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 08:14:23.422930 476 raft_consensus.cc:348] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.422948 479 raft_consensus.cc:348] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.423136 479 raft_consensus.cc:374] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:23.423179 476 raft_consensus.cc:374] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:23.423235 479 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.423269 476 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:23.423492 479 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.423491 476 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.424033 479 ts_tablet_manager.cc:1398] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:14:23.424037 476 ts_tablet_manager.cc:1398] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:14:23.439301 469 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:23.439491 469 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.440220 469 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:23.440837 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:23.440790 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:23.441061 32132 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:23.441076 32002 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:23.441531 31807 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:23.441797 469 raft_consensus.cc:2781] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:23.441910 469 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:23.441998 469 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:23.443850 469 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.444418 469 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:23.444841 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:23.444955 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:23.445040 32002 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:23.445159 32132 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:23.446985 32002 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:23.447180 32132 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:23.447366 31807 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:23.447651 469 raft_consensus.cc:2781] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:23.447788 469 raft_consensus.cc:686] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:23.448086 469 consensus_queue.cc:237] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.450678 31730 catalog_manager.cc:5484] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:23.595993 422 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:23.596235 422 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:23.597906 422 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:23.606175 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:23.606451 32132 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:23.606971 31938 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:23.606992 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:23.607431 32263 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:23.607445 422 raft_consensus.cc:2781] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:23.607563 422 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:23.607650 422 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:23.610059 422 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:23.610724 422 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:23.611177 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:23.611363 32132 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:23.611630 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:23.611810 32263 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:23.613202 32132 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:23.613584 31938 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:23.613983 422 raft_consensus.cc:2781] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:23.614157 32263 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:23.614794 422 raft_consensus.cc:686] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:23.615546 422 consensus_queue.cc:237] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:23.618722 31730 catalog_manager.cc:5484] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 1 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:23.765019 482 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:23.765262 482 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.766850 482 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:23.773152 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:23.773411 32002 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:23.773335 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:23.773566 32263 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:23.773898 32068 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:23.774257 482 raft_consensus.cc:2781] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:23.774381 482 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:23.774480 482 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:23.776494 482 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.777086 482 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:23.777565 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:23.777617 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:23.777770 32263 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:23.777815 32002 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:23.780063 32263 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:23.780284 32002 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:23.780539 32068 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:23.780895 482 raft_consensus.cc:2781] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:23.781275 482 raft_consensus.cc:686] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:23.781709 482 consensus_queue.cc:237] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:23.784963 31730 catalog_manager.cc:5484] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 1 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:23.843479 469 consensus_queue.cc:1035] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:23.849073 401 consensus_queue.cc:1035] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:24.081033 422 consensus_queue.cc:1035] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:24.087671 493 consensus_queue.cc:1035] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:24.172398 500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:24.172791 500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:24.180663 500 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 08:14:24.212568 482 consensus_queue.cc:1035] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:24.217958 482 consensus_queue.cc:1035] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 317.498 ms
time per row: 0.158749 ms
I20240501 08:14:24.649139 31728 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:24.657794 32243 tablet_service.cc:1467] Processing CreateTablet for tablet 4f80695b5a9f4832b1eac49f4eaeef5e (DEFAULT_TABLE table=rebalance_test_table_2 [id=616b0d8b0bbf41c2a26803d2d393e94a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:24.657791 31851 tablet_service.cc:1467] Processing CreateTablet for tablet 4f80695b5a9f4832b1eac49f4eaeef5e (DEFAULT_TABLE table=rebalance_test_table_2 [id=616b0d8b0bbf41c2a26803d2d393e94a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:24.658253 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f80695b5a9f4832b1eac49f4eaeef5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.658253 31851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f80695b5a9f4832b1eac49f4eaeef5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.659304 32113 tablet_service.cc:1467] Processing CreateTablet for tablet 2b96418c77b6483dba686be41d8f5934 (DEFAULT_TABLE table=rebalance_test_table_2 [id=616b0d8b0bbf41c2a26803d2d393e94a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:24.659477 32241 tablet_service.cc:1467] Processing CreateTablet for tablet d751eea1a7344c2f80ca2fa033d0ee64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=616b0d8b0bbf41c2a26803d2d393e94a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:24.659737 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b96418c77b6483dba686be41d8f5934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.659888 32241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d751eea1a7344c2f80ca2fa033d0ee64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.662492 31983 tablet_service.cc:1467] Processing CreateTablet for tablet 4f80695b5a9f4832b1eac49f4eaeef5e (DEFAULT_TABLE table=rebalance_test_table_2 [id=616b0d8b0bbf41c2a26803d2d393e94a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:24.663137 31981 tablet_service.cc:1467] Processing CreateTablet for tablet d751eea1a7344c2f80ca2fa033d0ee64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=616b0d8b0bbf41c2a26803d2d393e94a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:24.663724 32242 tablet_service.cc:1467] Processing CreateTablet for tablet 2b96418c77b6483dba686be41d8f5934 (DEFAULT_TABLE table=rebalance_test_table_2 [id=616b0d8b0bbf41c2a26803d2d393e94a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:24.664111 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b96418c77b6483dba686be41d8f5934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.664994 31850 tablet_service.cc:1467] Processing CreateTablet for tablet 2b96418c77b6483dba686be41d8f5934 (DEFAULT_TABLE table=rebalance_test_table_2 [id=616b0d8b0bbf41c2a26803d2d393e94a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:24.665139 31852 tablet_service.cc:1467] Processing CreateTablet for tablet d751eea1a7344c2f80ca2fa033d0ee64 (DEFAULT_TABLE table=rebalance_test_table_2 [id=616b0d8b0bbf41c2a26803d2d393e94a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:24.665408 31850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b96418c77b6483dba686be41d8f5934. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.665616 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d751eea1a7344c2f80ca2fa033d0ee64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.668829 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f80695b5a9f4832b1eac49f4eaeef5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.673385 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d751eea1a7344c2f80ca2fa033d0ee64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:24.674686 571 tablet_bootstrap.cc:492] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:24.676604 571 tablet_bootstrap.cc:654] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:24.682509 573 tablet_bootstrap.cc:492] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:24.686416 573 tablet_bootstrap.cc:654] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:24.688470 570 tablet_bootstrap.cc:492] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:24.689474 571 tablet_bootstrap.cc:492] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:24.689736 571 ts_tablet_manager.cc:1367] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20240501 08:14:24.690676 571 raft_consensus.cc:348] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:24.690934 571 raft_consensus.cc:374] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:24.691063 571 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.691350 571 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:24.692482 575 tablet_bootstrap.cc:492] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:24.693964 570 tablet_bootstrap.cc:654] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:24.694321 571 ts_tablet_manager.cc:1398] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 08:14:24.698554 573 tablet_bootstrap.cc:492] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:24.698765 573 ts_tablet_manager.cc:1367] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240501 08:14:24.699071 570 tablet_bootstrap.cc:492] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:24.699261 570 ts_tablet_manager.cc:1367] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.002s
I20240501 08:14:24.699774 575 tablet_bootstrap.cc:654] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:24.699680 573 raft_consensus.cc:348] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:24.699873 573 raft_consensus.cc:374] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:24.700029 573 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.700186 570 raft_consensus.cc:348] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:24.700464 570 raft_consensus.cc:374] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:24.700364 573 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:24.700599 570 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.700907 570 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:24.701133 573 ts_tablet_manager.cc:1398] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:24.701508 573 tablet_bootstrap.cc:492] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:24.703161 573 tablet_bootstrap.cc:654] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:24.703855 570 ts_tablet_manager.cc:1398] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:14:24.704223 570 tablet_bootstrap.cc:492] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:24.706391 570 tablet_bootstrap.cc:654] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:24.711454 573 tablet_bootstrap.cc:492] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:24.711741 573 ts_tablet_manager.cc:1367] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 08:14:24.712805 573 raft_consensus.cc:348] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:24.713196 573 raft_consensus.cc:374] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:24.713346 573 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.713769 573 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:24.714674 573 ts_tablet_manager.cc:1398] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:24.715167 573 tablet_bootstrap.cc:492] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:24.717372 573 tablet_bootstrap.cc:654] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:24.719605 570 tablet_bootstrap.cc:492] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:24.719897 570 ts_tablet_manager.cc:1367] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.008s
I20240501 08:14:24.720670 570 raft_consensus.cc:348] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:24.721037 570 raft_consensus.cc:374] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:24.721146 570 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.721448 570 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:24.722112 570 ts_tablet_manager.cc:1398] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:24.722437 570 tablet_bootstrap.cc:492] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:24.723964 570 tablet_bootstrap.cc:654] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:24.725454 573 tablet_bootstrap.cc:492] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:24.725806 573 ts_tablet_manager.cc:1367] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240501 08:14:24.727293 573 raft_consensus.cc:348] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:24.727669 573 raft_consensus.cc:374] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:24.727764 570 tablet_bootstrap.cc:492] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:24.727921 570 ts_tablet_manager.cc:1367] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 08:14:24.727945 573 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.728528 573 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:24.728708 570 raft_consensus.cc:348] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:24.728853 575 tablet_bootstrap.cc:492] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:24.728915 570 raft_consensus.cc:374] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:24.729018 575 ts_tablet_manager.cc:1367] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.037s user 0.012s sys 0.009s
I20240501 08:14:24.729015 570 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.729241 570 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:24.729900 570 ts_tablet_manager.cc:1398] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 08:14:24.730041 575 raft_consensus.cc:348] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:24.730104 573 ts_tablet_manager.cc:1398] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:14:24.730262 575 raft_consensus.cc:374] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:24.730352 575 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.730657 575 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:24.731333 575 ts_tablet_manager.cc:1398] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:24.731604 575 tablet_bootstrap.cc:492] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:24.733253 575 tablet_bootstrap.cc:654] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:24.737336 575 tablet_bootstrap.cc:492] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:24.737490 575 ts_tablet_manager.cc:1367] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240501 08:14:24.738065 575 raft_consensus.cc:348] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:24.738229 575 raft_consensus.cc:374] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:24.738296 575 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:24.738526 575 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:24.738994 575 ts_tablet_manager.cc:1398] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:14:24.814922 466 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:24.815114 466 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:24.815856 466 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:24.816414 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:24.816465 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:24.816653 32002 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:24.816727 31872 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:24.817149 32198 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:24.817420 466 raft_consensus.cc:2781] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:24.817518 466 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:24.817592 466 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:24.819463 466 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:24.820026 466 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:24.820457 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:24.820672 32002 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:24.820672 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:24.820883 31872 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:24.822861 32002 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:24.822955 31872 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:24.823280 32198 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:24.823567 466 raft_consensus.cc:2781] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:24.823954 466 raft_consensus.cc:686] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:24.824265 466 consensus_queue.cc:237] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:24.826789 31728 catalog_manager.cc:5484] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:24.891985 422 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:24.892171 422 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:24.892997 422 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:24.893451 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:24.893700 32263 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:24.894188 31938 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:24.894452 422 raft_consensus.cc:2781] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:24.894579 422 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:24.894711 422 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:24.897192 422 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:24.897975 422 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:24.898197 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:24.898425 32263 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:24.900085 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:24.900239 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:24.900336 31872 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:24.900938 32263 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:24.901288 31938 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:24.901590 422 raft_consensus.cc:2781] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:24.901711 422 raft_consensus.cc:686] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:24.901971 422 consensus_queue.cc:237] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:24.904731 31728 catalog_manager.cc:5484] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 1 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:24.922063 401 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:24.922214 401 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:24.922878 401 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:24.923405 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:24.923694 32132 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:24.924113 31807 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:24.924450 401 raft_consensus.cc:2781] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:24.924603 401 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:24.924724 401 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:24.927410 401 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:24.927976 401 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:24.928570 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:24.928799 32132 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:24.930454 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:24.930761 32263 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:24.930655 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:24.930939 32262 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:24.931509 32132 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:24.931895 31807 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:24.932191 401 raft_consensus.cc:2781] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:24.932381 401 raft_consensus.cc:686] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:24.932658 401 consensus_queue.cc:237] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:24.933586 32262 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:24.935343 31726 catalog_manager.cc:5484] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:25.215982 466 consensus_queue.cc:1035] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:25.221187 583 consensus_queue.cc:1035] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:25.303011 31871 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:25.305773 422 consensus_queue.cc:1035] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:25.310309 493 consensus_queue.cc:1035] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:25.353107 591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:25.353440 591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:25.361390 591 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 08:14:25.427495 401 consensus_queue.cc:1035] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:25.454540 401 consensus_queue.cc:1035] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:25.685679 31939 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.65:38515, user_credentials={real_user=slave}} blocked reactor thread for 54219.2us
INSERT report
rows total: 2000
time total: 306.92 ms
time per row: 0.15346 ms
I20240501 08:14:25.816141 31725 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:25.825393 31983 tablet_service.cc:1467] Processing CreateTablet for tablet 5aac3040ec2f4c2185a48967e3c245ee (DEFAULT_TABLE table=rebalance_test_table_3 [id=082362edb8d74f93ba2194d62b5d0e34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:25.825623 32242 tablet_service.cc:1467] Processing CreateTablet for tablet 40d7e0033de04a42809bac000fda2a81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=082362edb8d74f93ba2194d62b5d0e34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:25.825884 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aac3040ec2f4c2185a48967e3c245ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:25.826037 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d7e0033de04a42809bac000fda2a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:25.826411 32113 tablet_service.cc:1467] Processing CreateTablet for tablet 5aac3040ec2f4c2185a48967e3c245ee (DEFAULT_TABLE table=rebalance_test_table_3 [id=082362edb8d74f93ba2194d62b5d0e34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:25.826790 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aac3040ec2f4c2185a48967e3c245ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:25.828934 31981 tablet_service.cc:1467] Processing CreateTablet for tablet 1e872bea48fc479686691121187aa0db (DEFAULT_TABLE table=rebalance_test_table_3 [id=082362edb8d74f93ba2194d62b5d0e34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:25.829315 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e872bea48fc479686691121187aa0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:25.832194 31852 tablet_service.cc:1467] Processing CreateTablet for tablet 5aac3040ec2f4c2185a48967e3c245ee (DEFAULT_TABLE table=rebalance_test_table_3 [id=082362edb8d74f93ba2194d62b5d0e34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:25.832674 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aac3040ec2f4c2185a48967e3c245ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:25.832707 32111 tablet_service.cc:1467] Processing CreateTablet for tablet 40d7e0033de04a42809bac000fda2a81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=082362edb8d74f93ba2194d62b5d0e34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:25.832799 655 tablet_bootstrap.cc:492] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:25.833137 32111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d7e0033de04a42809bac000fda2a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:25.834697 655 tablet_bootstrap.cc:654] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:25.836761 32112 tablet_service.cc:1467] Processing CreateTablet for tablet 1e872bea48fc479686691121187aa0db (DEFAULT_TABLE table=rebalance_test_table_3 [id=082362edb8d74f93ba2194d62b5d0e34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:25.837298 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e872bea48fc479686691121187aa0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:25.840260 31851 tablet_service.cc:1467] Processing CreateTablet for tablet 40d7e0033de04a42809bac000fda2a81 (DEFAULT_TABLE table=rebalance_test_table_3 [id=082362edb8d74f93ba2194d62b5d0e34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:25.840901 31851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d7e0033de04a42809bac000fda2a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:25.841079 32243 tablet_service.cc:1467] Processing CreateTablet for tablet 1e872bea48fc479686691121187aa0db (DEFAULT_TABLE table=rebalance_test_table_3 [id=082362edb8d74f93ba2194d62b5d0e34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:25.840922 658 tablet_bootstrap.cc:492] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:25.841569 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e872bea48fc479686691121187aa0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:25.843526 658 tablet_bootstrap.cc:654] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:25.852203 655 tablet_bootstrap.cc:492] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:25.852433 655 ts_tablet_manager.cc:1367] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20240501 08:14:25.853205 655 raft_consensus.cc:348] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:25.853420 655 raft_consensus.cc:374] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:25.853922 655 raft_consensus.cc:729] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:25.854174 655 consensus_queue.cc:260] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:25.854743 655 ts_tablet_manager.cc:1398] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:25.864029 660 tablet_bootstrap.cc:492] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:25.864552 656 tablet_bootstrap.cc:492] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:25.864785 658 tablet_bootstrap.cc:492] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:25.864948 658 ts_tablet_manager.cc:1367] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.004s
I20240501 08:14:25.865828 658 raft_consensus.cc:348] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:25.866068 658 raft_consensus.cc:374] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:25.866164 658 raft_consensus.cc:729] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:25.866242 656 tablet_bootstrap.cc:654] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:25.866353 660 tablet_bootstrap.cc:654] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:25.866993 658 consensus_queue.cc:260] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:25.867709 658 ts_tablet_manager.cc:1398] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:14:25.867988 658 tablet_bootstrap.cc:492] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:25.869174 655 tablet_bootstrap.cc:492] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:25.869505 658 tablet_bootstrap.cc:654] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:25.872808 655 tablet_bootstrap.cc:654] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:25.875939 656 tablet_bootstrap.cc:492] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:25.876077 658 tablet_bootstrap.cc:492] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:25.876156 656 ts_tablet_manager.cc:1367] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240501 08:14:25.876281 658 ts_tablet_manager.cc:1367] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 08:14:25.877315 656 raft_consensus.cc:348] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:25.877342 658 raft_consensus.cc:348] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:25.877646 658 raft_consensus.cc:374] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:25.877672 656 raft_consensus.cc:374] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:25.877722 658 raft_consensus.cc:729] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:25.877792 656 raft_consensus.cc:729] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:25.878078 656 consensus_queue.cc:260] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:25.878088 658 consensus_queue.cc:260] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:25.878764 656 ts_tablet_manager.cc:1398] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:25.879060 656 tablet_bootstrap.cc:492] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:25.880906 658 ts_tablet_manager.cc:1398] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240501 08:14:25.881116 656 tablet_bootstrap.cc:654] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:25.882932 655 tablet_bootstrap.cc:492] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:25.883157 655 ts_tablet_manager.cc:1367] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240501 08:14:25.884196 655 raft_consensus.cc:348] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:25.884743 655 raft_consensus.cc:374] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:25.884846 655 raft_consensus.cc:729] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:25.885192 655 consensus_queue.cc:260] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:25.885885 655 ts_tablet_manager.cc:1398] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 08:14:25.886023 660 tablet_bootstrap.cc:492] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:25.886179 660 ts_tablet_manager.cc:1367] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.006s
I20240501 08:14:25.886255 655 tablet_bootstrap.cc:492] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:25.887811 660 raft_consensus.cc:348] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:25.887989 660 raft_consensus.cc:374] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:25.888080 660 raft_consensus.cc:729] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:25.888345 655 tablet_bootstrap.cc:654] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:25.888386 660 consensus_queue.cc:260] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:25.891216 656 tablet_bootstrap.cc:492] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:25.891458 656 ts_tablet_manager.cc:1367] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240501 08:14:25.891762 660 ts_tablet_manager.cc:1398] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:14:25.892097 660 tablet_bootstrap.cc:492] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:25.892488 656 raft_consensus.cc:348] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:25.892782 656 raft_consensus.cc:374] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:25.892884 656 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:25.893195 656 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:25.893890 656 ts_tablet_manager.cc:1398] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:25.894229 422 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:25.894356 655 tablet_bootstrap.cc:492] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:25.894491 655 ts_tablet_manager.cc:1367] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240501 08:14:25.894430 422 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:25.895103 422 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:25.895357 655 raft_consensus.cc:348] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:25.895460 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:25.895565 655 raft_consensus.cc:374] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:25.895670 655 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:25.895740 32132 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:25.895882 655 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:25.896155 31938 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:25.896131 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:25.896423 422 raft_consensus.cc:2781] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:25.896435 655 ts_tablet_manager.cc:1398] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:25.896423 31871 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:25.896524 422 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:25.896607 422 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:25.898859 422 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:25.899662 660 tablet_bootstrap.cc:654] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:25.899792 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:25.899992 32132 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:25.900790 422 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:25.901309 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:25.901516 31871 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:25.902014 660 tablet_bootstrap.cc:492] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:25.902165 660 ts_tablet_manager.cc:1367] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240501 08:14:25.902954 660 raft_consensus.cc:348] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:25.903215 660 raft_consensus.cc:374] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:25.903292 660 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:25.903604 660 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:25.904019 32132 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:25.904168 660 ts_tablet_manager.cc:1398] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:14:25.904393 31871 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:25.904832 31939 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:25.905129 422 raft_consensus.cc:2781] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:25.905287 422 raft_consensus.cc:686] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:25.905546 482 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:25.905580 422 consensus_queue.cc:237] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:25.905721 482 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:25.906395 482 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:25.906919 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:25.906975 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:25.907119 32002 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:25.907116 32262 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:25.907546 32068 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:25.907811 482 raft_consensus.cc:2781] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:25.907907 482 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:25.908017 482 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:25.908732 31725 catalog_manager.cc:5484] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 1 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:25.910394 482 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:25.910948 482 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:25.911339 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:25.911489 32002 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:25.911446 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:25.911661 32262 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:25.913846 32002 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:25.914142 32262 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:25.914228 32068 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:25.914537 482 raft_consensus.cc:2781] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:25.914948 482 raft_consensus.cc:686] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:25.915180 482 consensus_queue.cc:237] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:25.917501 31725 catalog_manager.cc:5484] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 1 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:26.005662 622 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:26.005856 622 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:26.006543 622 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:26.007169 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:26.007201 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:26.007397 32262 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:26.007426 32132 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:26.007845 31807 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:26.008129 622 raft_consensus.cc:2781] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:26.008230 622 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:26.008343 622 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:26.010334 622 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:26.010874 622 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:26.011381 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:26.011492 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:26.011571 32262 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:26.011685 32132 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:26.013648 32132 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:26.013684 32262 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:26.014050 31807 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:26.014340 622 raft_consensus.cc:2781] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:26.014477 622 raft_consensus.cc:686] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:26.014780 622 consensus_queue.cc:237] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:26.017639 31725 catalog_manager.cc:5484] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:26.316147 482 consensus_queue.cc:1035] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:26.320976 482 consensus_queue.cc:1035] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:26.402014 622 consensus_queue.cc:1035] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:26.403007 672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:26.403429 672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:14:26.406695 401 consensus_queue.cc:1035] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:26.413730 672 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 08:14:26.460357 422 consensus_queue.cc:1035] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:26.466717 422 consensus_queue.cc:1035] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 309.386 ms
time per row: 0.154693 ms
-----------------------------------------------------
Suppressions used:
count bytes template
28 7248 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:14:27.814023 31723 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:27.823324 31852 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8e84d1538e460ebb9785fca94288bd (DEFAULT_TABLE table=rebalance_test_table_4 [id=fa3f315ffc684fb3aabf6d415b7acabd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:27.823546 32111 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8e84d1538e460ebb9785fca94288bd (DEFAULT_TABLE table=rebalance_test_table_4 [id=fa3f315ffc684fb3aabf6d415b7acabd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:27.823843 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8e84d1538e460ebb9785fca94288bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:27.823982 32111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8e84d1538e460ebb9785fca94288bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:27.824108 31851 tablet_service.cc:1467] Processing CreateTablet for tablet cef61d27228f43cfbae61993aee7f432 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fa3f315ffc684fb3aabf6d415b7acabd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:27.824513 31851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef61d27228f43cfbae61993aee7f432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:27.828003 31983 tablet_service.cc:1467] Processing CreateTablet for tablet 1e8e84d1538e460ebb9785fca94288bd (DEFAULT_TABLE table=rebalance_test_table_4 [id=fa3f315ffc684fb3aabf6d415b7acabd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:27.828472 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e8e84d1538e460ebb9785fca94288bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:27.828436 32242 tablet_service.cc:1467] Processing CreateTablet for tablet cef61d27228f43cfbae61993aee7f432 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fa3f315ffc684fb3aabf6d415b7acabd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:27.828847 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef61d27228f43cfbae61993aee7f432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:27.828809 32243 tablet_service.cc:1467] Processing CreateTablet for tablet 61369fbeb41a49588907104bcfd6fa0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=fa3f315ffc684fb3aabf6d415b7acabd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:27.829180 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61369fbeb41a49588907104bcfd6fa0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:27.829855 31850 tablet_service.cc:1467] Processing CreateTablet for tablet 61369fbeb41a49588907104bcfd6fa0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=fa3f315ffc684fb3aabf6d415b7acabd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:27.830245 31850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61369fbeb41a49588907104bcfd6fa0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:27.828472 31981 tablet_service.cc:1467] Processing CreateTablet for tablet cef61d27228f43cfbae61993aee7f432 (DEFAULT_TABLE table=rebalance_test_table_4 [id=fa3f315ffc684fb3aabf6d415b7acabd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:27.833025 32112 tablet_service.cc:1467] Processing CreateTablet for tablet 61369fbeb41a49588907104bcfd6fa0f (DEFAULT_TABLE table=rebalance_test_table_4 [id=fa3f315ffc684fb3aabf6d415b7acabd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:27.833225 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef61d27228f43cfbae61993aee7f432. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:27.833410 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61369fbeb41a49588907104bcfd6fa0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:27.834898 740 tablet_bootstrap.cc:492] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:27.836545 740 tablet_bootstrap.cc:654] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:27.840886 742 tablet_bootstrap.cc:492] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:27.842701 742 tablet_bootstrap.cc:654] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:27.850929 742 tablet_bootstrap.cc:492] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:27.851220 742 ts_tablet_manager.cc:1367] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240501 08:14:27.852106 742 raft_consensus.cc:348] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.852445 742 raft_consensus.cc:374] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:27.852593 742 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:27.852948 742 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.856173 742 ts_tablet_manager.cc:1398] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:14:27.856345 743 tablet_bootstrap.cc:492] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:27.859414 742 tablet_bootstrap.cc:492] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:27.859668 743 tablet_bootstrap.cc:654] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:27.861137 742 tablet_bootstrap.cc:654] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:27.861138 739 tablet_bootstrap.cc:492] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:27.863037 739 tablet_bootstrap.cc:654] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:27.865906 740 tablet_bootstrap.cc:492] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:27.866073 742 tablet_bootstrap.cc:492] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:27.866155 740 ts_tablet_manager.cc:1367] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.031s user 0.016s sys 0.013s
I20240501 08:14:27.866400 742 ts_tablet_manager.cc:1367] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240501 08:14:27.867235 740 raft_consensus.cc:348] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.867372 742 raft_consensus.cc:348] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:27.867624 740 raft_consensus.cc:374] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:27.867673 742 raft_consensus.cc:374] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:27.867777 740 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:27.867920 742 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:27.868069 740 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.868340 743 tablet_bootstrap.cc:492] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:27.868536 743 ts_tablet_manager.cc:1367] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20240501 08:14:27.868903 740 ts_tablet_manager.cc:1398] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:14:27.869313 740 tablet_bootstrap.cc:492] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:27.869313 743 raft_consensus.cc:348] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:27.869524 743 raft_consensus.cc:374] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:27.869626 743 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:27.869885 743 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:27.871241 743 ts_tablet_manager.cc:1398] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:27.871627 743 tablet_bootstrap.cc:492] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:27.871915 740 tablet_bootstrap.cc:654] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:27.873493 743 tablet_bootstrap.cc:654] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:27.875752 739 tablet_bootstrap.cc:492] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:27.876048 739 ts_tablet_manager.cc:1367] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20240501 08:14:27.868326 742 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:27.877506 739 raft_consensus.cc:348] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.878055 742 ts_tablet_manager.cc:1398] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 08:14:27.879791 743 tablet_bootstrap.cc:492] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:27.879901 740 tablet_bootstrap.cc:492] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:27.879959 743 ts_tablet_manager.cc:1367] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 08:14:27.880117 740 ts_tablet_manager.cc:1367] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20240501 08:14:27.877805 739 raft_consensus.cc:374] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:27.880475 739 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:27.880760 739 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.880946 740 raft_consensus.cc:348] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.881197 740 raft_consensus.cc:374] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:27.881291 740 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:27.881556 739 ts_tablet_manager.cc:1398] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:14:27.881469 743 raft_consensus.cc:348] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.881559 740 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.881740 743 raft_consensus.cc:374] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:27.881853 743 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:27.882150 740 ts_tablet_manager.cc:1398] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:27.882160 743 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.882774 743 ts_tablet_manager.cc:1398] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:27.883616 739 tablet_bootstrap.cc:492] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:27.885417 739 tablet_bootstrap.cc:654] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:27.887802 739 tablet_bootstrap.cc:492] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:27.888134 739 ts_tablet_manager.cc:1367] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 08:14:27.888725 739 raft_consensus.cc:348] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.888914 739 raft_consensus.cc:374] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:27.888988 739 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:27.889217 739 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:27.889660 739 ts_tablet_manager.cc:1398] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 08:14:27.889916 739 tablet_bootstrap.cc:492] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:27.891052 739 tablet_bootstrap.cc:654] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:27.892834 739 tablet_bootstrap.cc:492] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:27.892978 739 ts_tablet_manager.cc:1367] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:14:27.893599 739 raft_consensus.cc:348] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:27.893797 739 raft_consensus.cc:374] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:27.893903 739 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:27.894095 739 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:27.894644 739 ts_tablet_manager.cc:1398] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:14:28.031774 717 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:28.032004 717 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:28.032676 717 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:28.033347 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:28.033350 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:28.033560 32133 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:28.033602 32002 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:28.034073 31807 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:28.034337 717 raft_consensus.cc:2781] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:28.034456 717 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:28.034551 717 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:28.036502 717 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:28.037086 717 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:28.037501 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:28.037560 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:28.037703 32133 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:28.037758 32002 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:28.039530 32133 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:28.039556 32002 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:28.039954 31807 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:28.040243 717 raft_consensus.cc:2781] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:28.040657 717 raft_consensus.cc:686] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:28.040957 717 consensus_queue.cc:237] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:28.043802 31728 catalog_manager.cc:5484] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:28.058269 717 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:28.058496 717 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:28.059125 717 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:28.059616 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:28.059849 32262 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:28.059937 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:28.060149 32002 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:28.060370 31807 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:28.060631 717 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:28.060719 717 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:28.060822 717 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:28.062754 717 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:28.063321 717 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:28.063813 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:28.064051 32262 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:28.064087 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:28.064316 32002 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:28.066088 32262 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:28.066535 31807 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:28.066809 717 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:28.066967 717 raft_consensus.cc:686] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:28.067165 32002 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:28.067215 717 consensus_queue.cc:237] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:28.070639 31728 catalog_manager.cc:5484] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:28.251657 482 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:28.251839 482 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:28.252597 482 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:28.253270 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:28.253516 32262 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:28.254114 32068 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:28.254422 482 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:28.254559 482 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:28.254673 482 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:28.256759 482 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:28.257402 482 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:28.257982 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:28.258175 32262 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:28.259407 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:28.259543 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:28.259642 31872 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:28.260275 32262 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:28.260686 32068 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:28.260984 482 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:28.261329 482 raft_consensus.cc:686] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:28.261591 482 consensus_queue.cc:237] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:28.264155 31728 catalog_manager.cc:5484] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 1 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:28.450037 717 consensus_queue.cc:1035] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:28.454689 752 consensus_queue.cc:1035] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:28.561411 752 consensus_queue.cc:1035] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:28.566246 717 consensus_queue.cc:1035] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:28.654141 759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:28.654469 759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:28.662040 759 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 08:14:28.821750 32262 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:14:28.823474 482 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:28.849733 31871 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:28.851902 31871 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:14:28.852582 482 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:28.926858 806 mvcc.cc:205] Tried to move back new op lower bound from 7022801997085614080 to 7022801994801455104. Current Snapshot: MvccSnapshot[applied={T|T < 7022801997085614080}]
INSERT report
rows total: 2000
time total: 242.447 ms
time per row: 0.121223 ms
I20240501 08:14:29.060151 31722 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:29.069320 32111 tablet_service.cc:1467] Processing CreateTablet for tablet b9fa92579c134968bc91552c0542b1fc (DEFAULT_TABLE table=rebalance_test_table_5 [id=80e84ee33c3a4e379ca2a64ae16d209e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:29.069509 31981 tablet_service.cc:1467] Processing CreateTablet for tablet b9fa92579c134968bc91552c0542b1fc (DEFAULT_TABLE table=rebalance_test_table_5 [id=80e84ee33c3a4e379ca2a64ae16d209e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:29.069784 32111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fa92579c134968bc91552c0542b1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.069991 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fa92579c134968bc91552c0542b1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.070336 32243 tablet_service.cc:1467] Processing CreateTablet for tablet b9fa92579c134968bc91552c0542b1fc (DEFAULT_TABLE table=rebalance_test_table_5 [id=80e84ee33c3a4e379ca2a64ae16d209e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:29.070760 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9fa92579c134968bc91552c0542b1fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.076974 31983 tablet_service.cc:1467] Processing CreateTablet for tablet 4f78273f47414fd199e593595e6aae23 (DEFAULT_TABLE table=rebalance_test_table_5 [id=80e84ee33c3a4e379ca2a64ae16d209e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:29.077327 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f78273f47414fd199e593595e6aae23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.081125 31852 tablet_service.cc:1467] Processing CreateTablet for tablet b50688fadbf64a588bb2a9cba14024f5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=80e84ee33c3a4e379ca2a64ae16d209e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:29.081552 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50688fadbf64a588bb2a9cba14024f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.084439 827 tablet_bootstrap.cc:492] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:29.086071 827 tablet_bootstrap.cc:654] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:29.086392 32112 tablet_service.cc:1467] Processing CreateTablet for tablet 4f78273f47414fd199e593595e6aae23 (DEFAULT_TABLE table=rebalance_test_table_5 [id=80e84ee33c3a4e379ca2a64ae16d209e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:29.086880 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f78273f47414fd199e593595e6aae23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.089416 32242 tablet_service.cc:1467] Processing CreateTablet for tablet 4f78273f47414fd199e593595e6aae23 (DEFAULT_TABLE table=rebalance_test_table_5 [id=80e84ee33c3a4e379ca2a64ae16d209e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:29.089814 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f78273f47414fd199e593595e6aae23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.092954 31982 tablet_service.cc:1467] Processing CreateTablet for tablet b50688fadbf64a588bb2a9cba14024f5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=80e84ee33c3a4e379ca2a64ae16d209e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:29.093385 31982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50688fadbf64a588bb2a9cba14024f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.127678 830 tablet_bootstrap.cc:492] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:29.129109 32112 tablet_service.cc:1467] Processing CreateTablet for tablet b50688fadbf64a588bb2a9cba14024f5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=80e84ee33c3a4e379ca2a64ae16d209e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:29.129601 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b50688fadbf64a588bb2a9cba14024f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:29.130630 830 tablet_bootstrap.cc:654] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:29.136349 832 tablet_bootstrap.cc:492] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:29.138095 832 tablet_bootstrap.cc:654] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:29.139065 830 tablet_bootstrap.cc:492] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:29.139529 830 ts_tablet_manager.cc:1367] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240501 08:14:29.140515 830 raft_consensus.cc:348] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.141767 830 raft_consensus.cc:374] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:29.142030 830 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.142467 830 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.143850 828 tablet_bootstrap.cc:492] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:29.144409 832 tablet_bootstrap.cc:492] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:29.144536 832 ts_tablet_manager.cc:1367] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240501 08:14:29.144969 830 ts_tablet_manager.cc:1398] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:14:29.145396 832 raft_consensus.cc:348] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:29.145722 832 raft_consensus.cc:374] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:29.145860 832 raft_consensus.cc:729] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.145859 828 tablet_bootstrap.cc:654] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:29.146018 830 tablet_bootstrap.cc:492] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:29.146219 832 consensus_queue.cc:260] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:29.147691 832 ts_tablet_manager.cc:1398] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:29.148483 827 tablet_bootstrap.cc:492] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:29.148718 827 ts_tablet_manager.cc:1367] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.064s user 0.022s sys 0.029s
I20240501 08:14:29.149209 830 tablet_bootstrap.cc:654] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:29.149669 827 raft_consensus.cc:348] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.149926 827 raft_consensus.cc:374] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:29.150065 827 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.150471 827 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.151576 828 tablet_bootstrap.cc:492] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:29.151942 828 ts_tablet_manager.cc:1367] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 08:14:29.152174 827 ts_tablet_manager.cc:1398] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:29.152573 827 tablet_bootstrap.cc:492] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:29.152961 828 raft_consensus.cc:348] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.153244 828 raft_consensus.cc:374] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:29.153396 828 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.154758 827 tablet_bootstrap.cc:654] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:29.155992 828 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.158918 828 ts_tablet_manager.cc:1398] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 08:14:29.161440 828 tablet_bootstrap.cc:492] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:29.163115 828 tablet_bootstrap.cc:654] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:29.163348 827 tablet_bootstrap.cc:492] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:29.163631 827 ts_tablet_manager.cc:1367] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20240501 08:14:29.164682 827 raft_consensus.cc:348] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.164958 827 raft_consensus.cc:374] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:29.165105 827 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.165505 827 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.168864 828 tablet_bootstrap.cc:492] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:29.168851 827 ts_tablet_manager.cc:1398] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:14:29.169024 828 ts_tablet_manager.cc:1367] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240501 08:14:29.169155 827 tablet_bootstrap.cc:492] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:29.169415 830 tablet_bootstrap.cc:492] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:29.169584 830 ts_tablet_manager.cc:1367] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.016s
I20240501 08:14:29.169796 828 raft_consensus.cc:348] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:29.170161 828 raft_consensus.cc:374] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:29.170274 828 raft_consensus.cc:729] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.170555 830 raft_consensus.cc:348] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.170542 828 consensus_queue.cc:260] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:29.170784 830 raft_consensus.cc:374] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:29.170877 827 tablet_bootstrap.cc:654] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:29.170893 830 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.171195 830 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.171734 830 ts_tablet_manager.cc:1398] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:14:29.172106 828 ts_tablet_manager.cc:1398] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 08:14:29.172436 828 tablet_bootstrap.cc:492] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:29.174271 827 tablet_bootstrap.cc:492] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:29.174432 827 ts_tablet_manager.cc:1367] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240501 08:14:29.174834 828 tablet_bootstrap.cc:654] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:29.175158 827 raft_consensus.cc:348] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:29.175371 827 raft_consensus.cc:374] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:29.175483 827 raft_consensus.cc:729] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.175747 827 consensus_queue.cc:260] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:29.176371 827 ts_tablet_manager.cc:1398] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:29.177587 828 tablet_bootstrap.cc:492] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:29.177729 828 ts_tablet_manager.cc:1367] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 08:14:29.178496 828 raft_consensus.cc:348] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.178699 828 raft_consensus.cc:374] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:29.178789 828 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:29.178993 828 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.179499 828 ts_tablet_manager.cc:1398] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:14:29.198330 466 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:29.198519 466 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.199085 466 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:29.199683 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:29.199656 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:29.199946 32133 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:29.199951 32003 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:29.200430 32198 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:29.200752 466 raft_consensus.cc:2781] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:29.200874 466 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:29.200961 466 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:29.203094 466 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.203610 466 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:29.204118 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:29.204154 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:29.204396 32133 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:29.204396 32003 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:29.207226 32003 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:29.207226 32133 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:29.207662 32198 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:29.207943 466 raft_consensus.cc:2781] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:29.208334 466 raft_consensus.cc:686] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:29.208632 466 consensus_queue.cc:237] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.211222 31723 catalog_manager.cc:5484] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:29.235287 702 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:29.235471 702 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:29.236188 702 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:29.236735 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:29.236765 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:29.236961 31871 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:29.236976 32133 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:29.237367 31939 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:29.237643 702 raft_consensus.cc:2781] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:29.237769 702 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:29.237857 702 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:29.239812 702 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:29.240363 702 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:29.240844 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:29.240844 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:29.241025 31871 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:29.241034 32133 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:29.242858 31871 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:29.243036 32133 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:29.243254 31939 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:29.243535 702 raft_consensus.cc:2781] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:29.243906 702 raft_consensus.cc:686] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:29.244203 702 consensus_queue.cc:237] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:29.246868 31723 catalog_manager.cc:5484] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 1 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:29.432495 482 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:29.432742 482 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.433292 482 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:29.433820 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:29.433913 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:29.434046 32003 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:29.434130 32262 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:29.434433 32068 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:29.434722 482 raft_consensus.cc:2781] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:29.434806 482 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:29.434911 482 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:29.436795 482 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.437368 482 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:29.437776 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:29.437826 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:29.437947 32003 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:29.437983 32262 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:29.439780 32262 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:29.439800 32003 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:29.440127 32068 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:29.440366 482 raft_consensus.cc:2781] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:29.440531 482 raft_consensus.cc:686] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:29.440775 482 consensus_queue.cc:237] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:29.443199 31723 catalog_manager.cc:5484] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 1 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:29.623926 466 consensus_queue.cc:1035] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:29.640693 849 consensus_queue.cc:1035] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:29.715458 702 consensus_queue.cc:1035] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:29.719828 702 consensus_queue.cc:1035] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:29.817114 845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:29.817535 845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:29.825793 845 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 08:14:29.857273 482 consensus_queue.cc:1035] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:29.862164 810 consensus_queue.cc:1035] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 290.72 ms
time per row: 0.14536 ms
I20240501 08:14:30.268357 31723 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:30.277047 31852 tablet_service.cc:1467] Processing CreateTablet for tablet d5eb3eb4e16a4caeb60c74af37968e9c (DEFAULT_TABLE table=rebalance_test_table_6 [id=73e7ed9860b34a7d9174a5a1282f1f97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:30.277570 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5eb3eb4e16a4caeb60c74af37968e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:30.278395 31981 tablet_service.cc:1467] Processing CreateTablet for tablet d5eb3eb4e16a4caeb60c74af37968e9c (DEFAULT_TABLE table=rebalance_test_table_6 [id=73e7ed9860b34a7d9174a5a1282f1f97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:30.278787 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5eb3eb4e16a4caeb60c74af37968e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:30.278986 32243 tablet_service.cc:1467] Processing CreateTablet for tablet 617ca585bf1845f78b450d28ac3f36fd (DEFAULT_TABLE table=rebalance_test_table_6 [id=73e7ed9860b34a7d9174a5a1282f1f97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:30.279403 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617ca585bf1845f78b450d28ac3f36fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:30.282213 32241 tablet_service.cc:1467] Processing CreateTablet for tablet 0448f8ee0f7a4932932247445b361d79 (DEFAULT_TABLE table=rebalance_test_table_6 [id=73e7ed9860b34a7d9174a5a1282f1f97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:30.282413 32112 tablet_service.cc:1467] Processing CreateTablet for tablet 617ca585bf1845f78b450d28ac3f36fd (DEFAULT_TABLE table=rebalance_test_table_6 [id=73e7ed9860b34a7d9174a5a1282f1f97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:30.282675 32241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0448f8ee0f7a4932932247445b361d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:30.282850 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617ca585bf1845f78b450d28ac3f36fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:30.284868 32242 tablet_service.cc:1467] Processing CreateTablet for tablet d5eb3eb4e16a4caeb60c74af37968e9c (DEFAULT_TABLE table=rebalance_test_table_6 [id=73e7ed9860b34a7d9174a5a1282f1f97]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:30.285439 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5eb3eb4e16a4caeb60c74af37968e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:30.289616 31850 tablet_service.cc:1467] Processing CreateTablet for tablet 617ca585bf1845f78b450d28ac3f36fd (DEFAULT_TABLE table=rebalance_test_table_6 [id=73e7ed9860b34a7d9174a5a1282f1f97]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:30.289686 31851 tablet_service.cc:1467] Processing CreateTablet for tablet 0448f8ee0f7a4932932247445b361d79 (DEFAULT_TABLE table=rebalance_test_table_6 [id=73e7ed9860b34a7d9174a5a1282f1f97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:30.291244 31982 tablet_service.cc:1467] Processing CreateTablet for tablet 0448f8ee0f7a4932932247445b361d79 (DEFAULT_TABLE table=rebalance_test_table_6 [id=73e7ed9860b34a7d9174a5a1282f1f97]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:30.291674 31982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0448f8ee0f7a4932932247445b361d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:30.292851 911 tablet_bootstrap.cc:492] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:30.294878 911 tablet_bootstrap.cc:654] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:30.295217 31850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617ca585bf1845f78b450d28ac3f36fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:30.300624 31851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0448f8ee0f7a4932932247445b361d79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:30.304818 913 tablet_bootstrap.cc:492] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:30.308149 913 tablet_bootstrap.cc:654] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:30.313409 916 tablet_bootstrap.cc:492] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:30.313688 912 tablet_bootstrap.cc:492] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:30.314950 913 tablet_bootstrap.cc:492] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:30.315181 913 ts_tablet_manager.cc:1367] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 08:14:30.315555 912 tablet_bootstrap.cc:654] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:30.316236 913 raft_consensus.cc:348] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:30.316627 913 raft_consensus.cc:374] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:30.316788 913 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:30.317144 913 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:30.317723 911 tablet_bootstrap.cc:492] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:30.317871 913 ts_tablet_manager.cc:1398] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:30.317929 911 ts_tablet_manager.cc:1367] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.008s
I20240501 08:14:30.318198 913 tablet_bootstrap.cc:492] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:30.318337 916 tablet_bootstrap.cc:654] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:30.318585 912 tablet_bootstrap.cc:492] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:30.318776 912 ts_tablet_manager.cc:1367] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:14:30.318958 911 raft_consensus.cc:348] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:30.319231 911 raft_consensus.cc:374] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:30.319370 911 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:30.319586 912 raft_consensus.cc:348] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:30.319847 912 raft_consensus.cc:374] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:30.319975 912 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:30.319965 911 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:30.320245 912 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:30.320881 911 ts_tablet_manager.cc:1398] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:30.320923 912 ts_tablet_manager.cc:1398] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:30.321235 912 tablet_bootstrap.cc:492] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:30.321286 911 tablet_bootstrap.cc:492] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:30.323046 911 tablet_bootstrap.cc:654] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:30.324479 912 tablet_bootstrap.cc:654] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:30.325287 913 tablet_bootstrap.cc:654] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:30.328363 911 tablet_bootstrap.cc:492] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:30.328673 911 ts_tablet_manager.cc:1367] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.004s
I20240501 08:14:30.329607 911 raft_consensus.cc:348] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:30.329934 911 raft_consensus.cc:374] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:30.330077 911 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:30.330423 911 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:30.331270 911 ts_tablet_manager.cc:1398] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:30.335363 916 tablet_bootstrap.cc:492] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:30.335559 916 ts_tablet_manager.cc:1367] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.005s
I20240501 08:14:30.336275 916 raft_consensus.cc:348] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:30.336489 916 raft_consensus.cc:374] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:30.336596 916 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:30.336836 916 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:30.337442 916 ts_tablet_manager.cc:1398] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:30.338908 913 tablet_bootstrap.cc:492] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:30.339241 913 ts_tablet_manager.cc:1367] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.004s
I20240501 08:14:30.339720 912 tablet_bootstrap.cc:492] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:30.339939 912 ts_tablet_manager.cc:1367] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.009s
I20240501 08:14:30.340073 913 raft_consensus.cc:348] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:30.340785 913 raft_consensus.cc:374] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:30.340708 912 raft_consensus.cc:348] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:30.340919 913 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:30.340972 912 raft_consensus.cc:374] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:30.341060 912 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:30.341315 912 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:30.341971 912 ts_tablet_manager.cc:1398] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:30.342278 912 tablet_bootstrap.cc:492] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:30.343238 913 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:30.343845 912 tablet_bootstrap.cc:654] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:30.344012 913 ts_tablet_manager.cc:1398] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 08:14:30.345782 913 tablet_bootstrap.cc:492] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:30.348282 912 tablet_bootstrap.cc:492] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:30.348320 913 tablet_bootstrap.cc:654] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:30.348477 912 ts_tablet_manager.cc:1367] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240501 08:14:30.349512 912 raft_consensus.cc:348] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:30.349711 912 raft_consensus.cc:374] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:30.349797 912 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:30.350039 912 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:30.350476 913 tablet_bootstrap.cc:492] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:30.350615 913 ts_tablet_manager.cc:1367] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240501 08:14:30.351229 912 ts_tablet_manager.cc:1398] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:30.351346 840 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:30.351475 840 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:30.352073 840 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:30.352370 913 raft_consensus.cc:348] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:30.352551 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:30.352627 913 raft_consensus.cc:374] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:30.352629 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:30.352732 913 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:30.352856 31871 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:30.352856 32002 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:30.352979 913 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:30.353348 32198 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:30.353682 913 ts_tablet_manager.cc:1398] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240501 08:14:30.353729 840 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:30.353863 840 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:30.353958 840 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:30.356206 840 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:30.356827 840 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:30.357496 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:30.357558 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:30.357693 31871 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:30.357779 32002 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:30.359764 32002 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:30.359788 31871 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:30.360158 32198 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:30.360457 840 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:30.360584 840 raft_consensus.cc:686] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:30.360957 840 consensus_queue.cc:237] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:30.363523 31723 catalog_manager.cc:5484] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:30.427829 842 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:30.427999 842 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:30.428717 842 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:30.429224 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:30.429208 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:30.429442 31871 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:30.429476 32263 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:30.429821 31939 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:30.430198 842 raft_consensus.cc:2781] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:30.430327 842 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:30.430429 842 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:30.432425 842 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:30.432931 842 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:30.433349 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:30.433413 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:30.433547 31871 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:30.433632 32263 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:30.436367 31871 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:30.436362 32263 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:30.436782 31939 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:30.437094 842 raft_consensus.cc:2781] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:30.437222 842 raft_consensus.cc:686] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:30.437470 842 consensus_queue.cc:237] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:30.440263 31723 catalog_manager.cc:5484] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 1 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:30.631904 849 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:30.632113 849 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:30.632748 849 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:30.633200 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:30.633453 32133 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:30.633445 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:30.633687 31871 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:30.633898 32198 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:30.634227 849 raft_consensus.cc:2781] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:30.634310 849 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:30.634433 849 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:30.636257 849 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:30.636933 849 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:30.637260 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:30.637470 32133 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:30.637517 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:30.637686 31871 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:30.639431 32133 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:30.639845 32198 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:30.640092 849 raft_consensus.cc:2781] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:30.640177 31871 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:30.640237 849 raft_consensus.cc:686] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:30.640580 849 consensus_queue.cc:237] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:30.643754 31723 catalog_manager.cc:5484] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:30.746328 849 consensus_queue.cc:1035] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:30.752884 849 consensus_queue.cc:1035] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:30.923529 702 consensus_queue.cc:1035] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:30.929095 842 consensus_queue.cc:1035] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:31.011584 927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:31.011937 927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:31.020416 927 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 08:14:31.176674 31871 raft_consensus.cc:1261] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:14:31.177338 849 consensus_queue.cc:1035] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:31.177515 32133 raft_consensus.cc:1261] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 08:14:31.178078 849 consensus_queue.cc:1035] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:31.224406 970 mvcc.cc:205] Tried to move back new op lower bound from 7022802006733869056 to 7022802004546379776. Current Snapshot: MvccSnapshot[applied={T|T < 7022802006733869056}]
I20240501 08:14:31.243168 929 mvcc.cc:205] Tried to move back new op lower bound from 7022802006733869056 to 7022802004546379776. Current Snapshot: MvccSnapshot[applied={T|T < 7022802006733869056 or (T in {7022802006733869056})}]
INSERT report
rows total: 2000
time total: 210.342 ms
time per row: 0.105171 ms
I20240501 08:14:31.393806 31727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:31.402778 32112 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1f781d8e3645e0b35d0be82f6880cf (DEFAULT_TABLE table=rebalance_test_table_7 [id=2550f280c7124157a44c5708738cc9c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:31.403285 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1f781d8e3645e0b35d0be82f6880cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:31.403337 32111 tablet_service.cc:1467] Processing CreateTablet for tablet b8011eb1bd4c42ce89517736009576e3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2550f280c7124157a44c5708738cc9c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:31.403717 32111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8011eb1bd4c42ce89517736009576e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:31.404740 31981 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1f781d8e3645e0b35d0be82f6880cf (DEFAULT_TABLE table=rebalance_test_table_7 [id=2550f280c7124157a44c5708738cc9c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:31.405143 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1f781d8e3645e0b35d0be82f6880cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:31.405601 32242 tablet_service.cc:1467] Processing CreateTablet for tablet b8011eb1bd4c42ce89517736009576e3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2550f280c7124157a44c5708738cc9c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:31.406033 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8011eb1bd4c42ce89517736009576e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:31.407644 31852 tablet_service.cc:1467] Processing CreateTablet for tablet 9c1f781d8e3645e0b35d0be82f6880cf (DEFAULT_TABLE table=rebalance_test_table_7 [id=2550f280c7124157a44c5708738cc9c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:31.407784 31982 tablet_service.cc:1467] Processing CreateTablet for tablet 5634d60715fc486b81d38f1dbad590e4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2550f280c7124157a44c5708738cc9c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:31.408047 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c1f781d8e3645e0b35d0be82f6880cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:31.408146 31982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5634d60715fc486b81d38f1dbad590e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:31.408653 31851 tablet_service.cc:1467] Processing CreateTablet for tablet b8011eb1bd4c42ce89517736009576e3 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2550f280c7124157a44c5708738cc9c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:31.409302 32113 tablet_service.cc:1467] Processing CreateTablet for tablet 5634d60715fc486b81d38f1dbad590e4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2550f280c7124157a44c5708738cc9c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:31.409646 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5634d60715fc486b81d38f1dbad590e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:31.412736 32243 tablet_service.cc:1467] Processing CreateTablet for tablet 5634d60715fc486b81d38f1dbad590e4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=2550f280c7124157a44c5708738cc9c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:31.413149 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5634d60715fc486b81d38f1dbad590e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:31.418218 31851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8011eb1bd4c42ce89517736009576e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:31.422736 988 tablet_bootstrap.cc:492] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:31.423729 990 tablet_bootstrap.cc:492] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:31.425750 988 tablet_bootstrap.cc:654] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:31.425858 991 tablet_bootstrap.cc:492] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:31.427582 990 tablet_bootstrap.cc:654] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:31.428874 991 tablet_bootstrap.cc:654] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:31.432808 990 tablet_bootstrap.cc:492] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:31.433022 990 ts_tablet_manager.cc:1367] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240501 08:14:31.433956 990 raft_consensus.cc:348] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:31.434337 990 raft_consensus.cc:374] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:31.434515 990 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:31.434979 990 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:31.436826 990 ts_tablet_manager.cc:1398] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:14:31.441555 990 tablet_bootstrap.cc:492] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:31.442473 991 tablet_bootstrap.cc:492] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:31.442723 989 tablet_bootstrap.cc:492] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:31.442932 991 ts_tablet_manager.cc:1367] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20240501 08:14:31.443552 990 tablet_bootstrap.cc:654] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:31.444002 988 tablet_bootstrap.cc:492] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:31.443993 991 raft_consensus.cc:348] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.444202 988 ts_tablet_manager.cc:1367] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20240501 08:14:31.444252 991 raft_consensus.cc:374] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:31.444412 991 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:31.444700 989 tablet_bootstrap.cc:654] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:31.444728 991 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.445156 988 raft_consensus.cc:348] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.445412 988 raft_consensus.cc:374] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:31.445539 988 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:31.445686 991 ts_tablet_manager.cc:1398] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:31.445912 988 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.446832 988 ts_tablet_manager.cc:1398] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:31.447278 988 tablet_bootstrap.cc:492] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:31.449328 988 tablet_bootstrap.cc:654] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:31.453552 990 tablet_bootstrap.cc:492] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:31.453797 990 ts_tablet_manager.cc:1367] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 08:14:31.454668 990 raft_consensus.cc:348] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.454972 990 raft_consensus.cc:374] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:31.455092 990 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:31.455423 990 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.456990 991 tablet_bootstrap.cc:492] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:31.457095 990 ts_tablet_manager.cc:1398] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:31.458422 991 tablet_bootstrap.cc:654] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:31.459573 989 tablet_bootstrap.cc:492] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:31.459795 989 ts_tablet_manager.cc:1367] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.003s
I20240501 08:14:31.460234 988 tablet_bootstrap.cc:492] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:31.460561 988 ts_tablet_manager.cc:1367] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20240501 08:14:31.460846 989 raft_consensus.cc:348] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:31.461131 989 raft_consensus.cc:374] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:31.461267 989 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:31.461369 988 raft_consensus.cc:348] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.461623 988 raft_consensus.cc:374] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:31.461736 988 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:31.461648 989 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:31.462003 988 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.462559 989 ts_tablet_manager.cc:1398] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:31.462591 988 ts_tablet_manager.cc:1398] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:14:31.462919 989 tablet_bootstrap.cc:492] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:31.464589 989 tablet_bootstrap.cc:654] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:31.465165 991 tablet_bootstrap.cc:492] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:31.465370 991 ts_tablet_manager.cc:1367] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20240501 08:14:31.466382 991 raft_consensus.cc:348] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.466735 991 raft_consensus.cc:374] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:31.466864 991 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:31.467166 991 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.467734 991 ts_tablet_manager.cc:1398] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 08:14:31.468075 991 tablet_bootstrap.cc:492] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:31.468127 989 tablet_bootstrap.cc:492] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:31.468343 989 ts_tablet_manager.cc:1367] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:14:31.469151 989 raft_consensus.cc:348] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.469442 991 tablet_bootstrap.cc:654] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:31.469460 989 raft_consensus.cc:374] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:31.469619 989 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:31.469895 989 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.471385 989 ts_tablet_manager.cc:1398] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:14:31.472635 991 tablet_bootstrap.cc:492] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:31.472790 991 ts_tablet_manager.cc:1367] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:14:31.473511 991 raft_consensus.cc:348] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:31.473711 991 raft_consensus.cc:374] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:31.473817 991 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:31.474043 991 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:31.474712 991 ts_tablet_manager.cc:1398] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:31.689075 1001 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:31.689271 1001 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:31.689949 1001 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:31.690555 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:31.690599 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:31.690831 31872 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:31.690847 32002 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:31.691427 32069 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:31.691740 1001 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:31.691866 1001 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:31.691965 1001 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:31.694242 1001 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:31.694823 1001 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:31.695230 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:31.695261 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:31.695453 32002 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:31.695493 31872 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:31.697619 31872 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:31.697633 32002 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:31.698081 32069 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:31.698457 1001 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:31.698593 1001 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:31.698871 1001 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:31.701581 31723 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 1 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:31.719157 1001 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:31.719365 1001 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.719931 1001 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:31.720420 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:31.720516 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:31.720715 32263 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:31.720705 31872 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:31.721097 32069 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:31.721365 1001 raft_consensus.cc:2781] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:31.721457 1001 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:31.721565 1001 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:31.723480 1001 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.724086 1001 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:31.724567 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:31.724767 31872 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:31.724750 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:31.724936 32263 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:31.726666 31872 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:31.726857 32263 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:31.727053 32069 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:31.727384 1001 raft_consensus.cc:2781] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:31.727545 1001 raft_consensus.cc:686] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:31.727813 1001 consensus_queue.cc:237] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.730765 31723 catalog_manager.cc:5484] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 1 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:31.735872 1001 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:31.736053 1001 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.736654 1001 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:31.737151 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:31.737155 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:31.737362 32002 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:31.737375 32263 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:31.737766 32068 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:31.738027 1001 raft_consensus.cc:2781] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:31.738132 1001 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:31.738222 1001 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:31.740051 1001 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.740597 1001 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:31.741081 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:31.741147 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:31.741295 32002 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:31.741335 32263 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:31.743125 32263 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:31.743125 32002 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:31.743510 32068 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:31.743798 1001 raft_consensus.cc:2781] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:31.743955 1001 raft_consensus.cc:686] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:31.744202 1001 consensus_queue.cc:237] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:31.747105 31723 catalog_manager.cc:5484] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 1 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:32.104164 1001 consensus_queue.cc:1035] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:32.129192 1001 consensus_queue.cc:1035] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:32.130939 1007 consensus_queue.cc:1035] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:32.139120 1009 consensus_queue.cc:1035] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:32.153869 1001 consensus_queue.cc:1035] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:32.168416 1001 consensus_queue.cc:1035] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:32.178642 1005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:32.179019 1005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:32.186807 1005 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 181.305 ms
time per row: 0.0906523 ms
-----------------------------------------------------
Suppressions used:
count bytes template
14 3624 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:14:33.460682 31727 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:33.470093 31981 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7795f40f344b2e92dfc60a5ff1173a (DEFAULT_TABLE table=rebalance_test_table_8 [id=feb63264fd0e434c892167c21faba43a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:33.470343 32243 tablet_service.cc:1467] Processing CreateTablet for tablet 73d13200a9464963bf7501faad8cd6f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=feb63264fd0e434c892167c21faba43a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:33.470573 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7795f40f344b2e92dfc60a5ff1173a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:33.470732 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d13200a9464963bf7501faad8cd6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:33.470620 32112 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7795f40f344b2e92dfc60a5ff1173a (DEFAULT_TABLE table=rebalance_test_table_8 [id=feb63264fd0e434c892167c21faba43a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:33.471056 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7795f40f344b2e92dfc60a5ff1173a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:33.471874 32242 tablet_service.cc:1467] Processing CreateTablet for tablet 90faa3f785f24c53a103ea93a3f405e0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=feb63264fd0e434c892167c21faba43a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:33.472247 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90faa3f785f24c53a103ea93a3f405e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:33.472679 31851 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7795f40f344b2e92dfc60a5ff1173a (DEFAULT_TABLE table=rebalance_test_table_8 [id=feb63264fd0e434c892167c21faba43a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:33.473064 31851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7795f40f344b2e92dfc60a5ff1173a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:33.477030 32113 tablet_service.cc:1467] Processing CreateTablet for tablet 90faa3f785f24c53a103ea93a3f405e0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=feb63264fd0e434c892167c21faba43a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:33.477038 31982 tablet_service.cc:1467] Processing CreateTablet for tablet 73d13200a9464963bf7501faad8cd6f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=feb63264fd0e434c892167c21faba43a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:33.477396 31982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d13200a9464963bf7501faad8cd6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:33.477406 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90faa3f785f24c53a103ea93a3f405e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:33.477835 1068 tablet_bootstrap.cc:492] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:33.479314 1068 tablet_bootstrap.cc:654] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:33.481532 31850 tablet_service.cc:1467] Processing CreateTablet for tablet 90faa3f785f24c53a103ea93a3f405e0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=feb63264fd0e434c892167c21faba43a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:33.481604 31852 tablet_service.cc:1467] Processing CreateTablet for tablet 73d13200a9464963bf7501faad8cd6f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=feb63264fd0e434c892167c21faba43a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:33.481966 31850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90faa3f785f24c53a103ea93a3f405e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:33.482172 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d13200a9464963bf7501faad8cd6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:33.498114 1070 tablet_bootstrap.cc:492] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:33.499511 1071 tablet_bootstrap.cc:492] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:33.500993 1070 tablet_bootstrap.cc:654] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:33.501119 1071 tablet_bootstrap.cc:654] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:33.503347 1071 tablet_bootstrap.cc:492] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:33.503430 1068 tablet_bootstrap.cc:492] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:33.503566 1071 ts_tablet_manager.cc:1367] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 08:14:33.503624 1068 ts_tablet_manager.cc:1367] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.008s
I20240501 08:14:33.504181 1072 tablet_bootstrap.cc:492] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:33.504652 1068 raft_consensus.cc:348] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:33.504716 1071 raft_consensus.cc:348] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:33.504921 1071 raft_consensus.cc:374] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:33.504921 1068 raft_consensus.cc:374] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:33.505009 1071 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:33.505029 1068 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:33.505228 1071 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:33.505286 1068 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:33.506055 1072 tablet_bootstrap.cc:654] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:33.508085 1068 ts_tablet_manager.cc:1398] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:14:33.508491 1068 tablet_bootstrap.cc:492] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:33.509729 1071 ts_tablet_manager.cc:1398] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 08:14:33.511188 1071 tablet_bootstrap.cc:492] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:33.511202 1068 tablet_bootstrap.cc:654] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:33.512573 1070 tablet_bootstrap.cc:492] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:33.512874 1070 ts_tablet_manager.cc:1367] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20240501 08:14:33.513309 1072 tablet_bootstrap.cc:492] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:33.513522 1072 ts_tablet_manager.cc:1367] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 08:14:33.514318 1072 raft_consensus.cc:348] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:33.514504 1072 raft_consensus.cc:374] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:33.514611 1072 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:33.514838 1071 tablet_bootstrap.cc:654] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:33.514825 1072 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:33.515385 1072 ts_tablet_manager.cc:1398] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:33.515769 1072 tablet_bootstrap.cc:492] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:33.513886 1070 raft_consensus.cc:348] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:33.516047 1070 raft_consensus.cc:374] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:33.516167 1070 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:33.516521 1070 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:33.517299 1070 ts_tablet_manager.cc:1398] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 08:14:33.517743 1070 tablet_bootstrap.cc:492] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:33.518616 1072 tablet_bootstrap.cc:654] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:33.519914 1070 tablet_bootstrap.cc:654] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:33.521857 1068 tablet_bootstrap.cc:492] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:33.522122 1068 ts_tablet_manager.cc:1367] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240501 08:14:33.523005 1068 raft_consensus.cc:348] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:33.523322 1068 raft_consensus.cc:374] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:33.523448 1068 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:33.523607 1071 tablet_bootstrap.cc:492] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:33.523792 1071 ts_tablet_manager.cc:1367] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 08:14:33.523797 1068 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:33.524470 1068 ts_tablet_manager.cc:1398] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:14:33.524591 1071 raft_consensus.cc:348] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:33.524812 1071 raft_consensus.cc:374] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:33.524881 1071 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:33.525141 1071 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:33.525698 1071 ts_tablet_manager.cc:1398] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 08:14:33.526433 1070 tablet_bootstrap.cc:492] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:33.527588 1070 ts_tablet_manager.cc:1367] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20240501 08:14:33.527706 1072 tablet_bootstrap.cc:492] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:33.528389 1070 raft_consensus.cc:348] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:33.528573 1072 ts_tablet_manager.cc:1367] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20240501 08:14:33.528656 1070 raft_consensus.cc:374] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:33.528755 1070 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:33.529012 1070 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:33.529336 1072 raft_consensus.cc:348] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:33.529569 1070 ts_tablet_manager.cc:1398] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:33.529580 1072 raft_consensus.cc:374] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:33.529654 1072 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:33.529911 1070 tablet_bootstrap.cc:492] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:33.530157 1072 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:33.530726 1072 ts_tablet_manager.cc:1398] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 08:14:33.531875 1070 tablet_bootstrap.cc:654] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:33.533746 1070 tablet_bootstrap.cc:492] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:33.533867 1070 ts_tablet_manager.cc:1367] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 08:14:33.534471 1070 raft_consensus.cc:348] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:33.534670 1070 raft_consensus.cc:374] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:33.534767 1070 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:33.535017 1070 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:33.535511 1070 ts_tablet_manager.cc:1398] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:14:33.661168 840 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:33.661357 840 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:33.661916 840 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:33.662423 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:33.662441 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:33.662680 32133 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:33.662693 31871 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:33.663141 32199 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:33.663435 840 raft_consensus.cc:2781] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:33.663540 840 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:33.663650 840 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:33.666301 840 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:33.666929 840 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:33.667213 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:33.667274 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:33.667418 32132 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:33.667487 31871 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:33.669267 32132 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:33.669471 31871 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:33.669899 32199 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:33.670284 840 raft_consensus.cc:2781] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:33.670451 840 raft_consensus.cc:686] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:33.670706 840 consensus_queue.cc:237] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:33.673734 31727 catalog_manager.cc:5484] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:33.713408 1062 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:33.713690 1062 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:33.714396 1062 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:33.715057 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:33.715057 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:33.715327 32262 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:33.715327 32003 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:33.715876 31807 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:33.716166 1062 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:33.716334 1062 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:33.716437 1062 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:33.719914 1062 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:33.720664 1062 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:33.721249 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:33.721491 32003 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:33.723201 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:33.723444 32262 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:33.724110 32003 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:33.724570 31807 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:33.724934 1062 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:33.725144 1062 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:33.725445 1062 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:33.725692 32262 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:33.729023 31727 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:33.822072 1062 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:33.822311 1062 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:33.822860 1062 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:33.823561 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:33.823607 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:33.823789 32133 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:33.823820 32003 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 0.
I20240501 08:14:33.824229 31807 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:33.824559 1062 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:33.824648 1062 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:33.824755 1062 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:33.826700 1062 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:33.827296 1062 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:33.827813 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:33.827873 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:33.828011 32003 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:33.828061 32133 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:33.829914 32133 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:33.829923 32003 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 1.
I20240501 08:14:33.830303 31807 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:33.830588 1062 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:33.830750 1062 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:33.831040 1062 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:33.833854 31727 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 1 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:34.142670 840 consensus_queue.cc:1035] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:34.150377 840 consensus_queue.cc:1035] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:34.212937 1086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:34.213323 1086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 08:14:34.218394 1081 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:34.223577 1081 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:34.224745 1086 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 08:14:34.291576 1062 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:34.308116 1097 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:34.561349 32068 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.56.66:34121, user_credentials={real_user=slave}} blocked reactor thread for 99407.7us
INSERT report
rows total: 2000
time total: 292.472 ms
time per row: 0.146236 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:14:35.635461 31731 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49838:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 08:14:35.645504 32113 tablet_service.cc:1467] Processing CreateTablet for tablet 84680e97e91b4e16906adc6a61d2ef9a (DEFAULT_TABLE table=rebalance_test_table_9 [id=027785a6e11447c8a7428a379b8ab95f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:35.646097 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84680e97e91b4e16906adc6a61d2ef9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:35.647203 31852 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe2118801714da99fe7f3c0585e57fc (DEFAULT_TABLE table=rebalance_test_table_9 [id=027785a6e11447c8a7428a379b8ab95f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:35.647666 31852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe2118801714da99fe7f3c0585e57fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:35.653144 32242 tablet_service.cc:1467] Processing CreateTablet for tablet 84680e97e91b4e16906adc6a61d2ef9a (DEFAULT_TABLE table=rebalance_test_table_9 [id=027785a6e11447c8a7428a379b8ab95f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:35.653429 31982 tablet_service.cc:1467] Processing CreateTablet for tablet 84680e97e91b4e16906adc6a61d2ef9a (DEFAULT_TABLE table=rebalance_test_table_9 [id=027785a6e11447c8a7428a379b8ab95f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:35.653589 32242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84680e97e91b4e16906adc6a61d2ef9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:35.653858 31982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84680e97e91b4e16906adc6a61d2ef9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:35.656561 1152 tablet_bootstrap.cc:492] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:35.656721 31983 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe2118801714da99fe7f3c0585e57fc (DEFAULT_TABLE table=rebalance_test_table_9 [id=027785a6e11447c8a7428a379b8ab95f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:35.657025 31983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe2118801714da99fe7f3c0585e57fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:35.658202 1152 tablet_bootstrap.cc:654] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:35.660547 32243 tablet_service.cc:1467] Processing CreateTablet for tablet ccfa1910629f47c6b3b6caf6c1a8c3f1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=027785a6e11447c8a7428a379b8ab95f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:35.660781 31981 tablet_service.cc:1467] Processing CreateTablet for tablet ccfa1910629f47c6b3b6caf6c1a8c3f1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=027785a6e11447c8a7428a379b8ab95f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:35.660987 32243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccfa1910629f47c6b3b6caf6c1a8c3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:35.661213 31981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccfa1910629f47c6b3b6caf6c1a8c3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:35.666141 32112 tablet_service.cc:1467] Processing CreateTablet for tablet 9fe2118801714da99fe7f3c0585e57fc (DEFAULT_TABLE table=rebalance_test_table_9 [id=027785a6e11447c8a7428a379b8ab95f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:35.666538 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fe2118801714da99fe7f3c0585e57fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:35.673900 1155 tablet_bootstrap.cc:492] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:35.678696 1155 tablet_bootstrap.cc:654] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:35.682806 32113 tablet_service.cc:1467] Processing CreateTablet for tablet ccfa1910629f47c6b3b6caf6c1a8c3f1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=027785a6e11447c8a7428a379b8ab95f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 08:14:35.683235 1156 tablet_bootstrap.cc:492] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97: Bootstrap starting.
I20240501 08:14:35.683440 32113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccfa1910629f47c6b3b6caf6c1a8c3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 08:14:35.686931 1155 tablet_bootstrap.cc:492] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:35.687678 1155 ts_tablet_manager.cc:1367] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240501 08:14:35.687747 1156 tablet_bootstrap.cc:654] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:35.689085 1155 raft_consensus.cc:348] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.689249 1154 tablet_bootstrap.cc:492] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:35.689636 1155 raft_consensus.cc:374] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:35.689817 1155 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:35.690109 1155 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.690872 1155 ts_tablet_manager.cc:1398] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:35.691200 1155 tablet_bootstrap.cc:492] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4: Bootstrap starting.
I20240501 08:14:35.692870 1155 tablet_bootstrap.cc:654] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:35.692873 1152 tablet_bootstrap.cc:492] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:35.693018 1152 ts_tablet_manager.cc:1367] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.016s
I20240501 08:14:35.694993 1152 raft_consensus.cc:348] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.695242 1152 raft_consensus.cc:374] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:35.695379 1152 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:35.695533 1154 tablet_bootstrap.cc:654] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:35.695596 1155 tablet_bootstrap.cc:492] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4: No bootstrap required, opened a new log
I20240501 08:14:35.695657 1152 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.696313 1152 ts_tablet_manager.cc:1398] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 08:14:35.695847 1155 ts_tablet_manager.cc:1367] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 08:14:35.697863 1152 tablet_bootstrap.cc:492] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:35.697695 1155 raft_consensus.cc:348] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.697979 1155 raft_consensus.cc:374] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:35.698108 1155 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Initialized, Role: FOLLOWER
I20240501 08:14:35.698453 1155 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.700323 1155 ts_tablet_manager.cc:1398] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 08:14:35.701041 1152 tablet_bootstrap.cc:654] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:35.701709 1156 tablet_bootstrap.cc:492] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97: No bootstrap required, opened a new log
I20240501 08:14:35.701928 1156 ts_tablet_manager.cc:1367] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20240501 08:14:35.703028 1156 raft_consensus.cc:348] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:35.703325 1156 raft_consensus.cc:374] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:35.703464 1156 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Initialized, Role: FOLLOWER
I20240501 08:14:35.703747 1156 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:35.704572 1156 ts_tablet_manager.cc:1398] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:14:35.707489 1152 tablet_bootstrap.cc:492] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:35.707798 1152 ts_tablet_manager.cc:1367] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20240501 08:14:35.708950 1152 raft_consensus.cc:348] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:35.709215 1152 raft_consensus.cc:374] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:35.709304 1152 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:35.709548 1152 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:35.709787 1154 tablet_bootstrap.cc:492] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:35.709954 1154 ts_tablet_manager.cc:1367] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.006s
I20240501 08:14:35.710131 1152 ts_tablet_manager.cc:1398] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:14:35.710525 1152 tablet_bootstrap.cc:492] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782: Bootstrap starting.
I20240501 08:14:35.710762 1154 raft_consensus.cc:348] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:35.710974 1154 raft_consensus.cc:374] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:35.711107 1154 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:35.711396 1154 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:35.711961 1154 ts_tablet_manager.cc:1398] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 08:14:35.712199 1152 tablet_bootstrap.cc:654] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:35.712455 1154 tablet_bootstrap.cc:492] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:35.714538 1154 tablet_bootstrap.cc:654] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:35.717036 1154 tablet_bootstrap.cc:492] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:35.717125 1152 tablet_bootstrap.cc:492] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782: No bootstrap required, opened a new log
I20240501 08:14:35.717196 1154 ts_tablet_manager.cc:1367] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 08:14:35.717288 1152 ts_tablet_manager.cc:1367] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240501 08:14:35.717952 1154 raft_consensus.cc:348] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.717993 1152 raft_consensus.cc:348] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.718132 1154 raft_consensus.cc:374] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:35.718137 1152 raft_consensus.cc:374] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:35.718212 1154 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:35.718223 1152 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Initialized, Role: FOLLOWER
I20240501 08:14:35.718448 1152 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.718478 1154 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.718946 1154 ts_tablet_manager.cc:1398] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 08:14:35.718950 1152 ts_tablet_manager.cc:1398] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 08:14:35.719209 1154 tablet_bootstrap.cc:492] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf: Bootstrap starting.
I20240501 08:14:35.719640 1146 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:35.719791 1146 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:35.720566 1146 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:35.720942 1154 tablet_bootstrap.cc:654] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf: Neither blocks nor log segments found. Creating new log.
I20240501 08:14:35.721292 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97" is_pre_election: true
I20240501 08:14:35.721550 31872 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:35.722159 31939 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:35.723121 1146 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:35.723151 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:35.723279 1146 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:35.723330 32133 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 0.
I20240501 08:14:35.723394 1146 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:35.726868 1146 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:35.727178 1154 tablet_bootstrap.cc:492] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf: No bootstrap required, opened a new log
I20240501 08:14:35.727355 1154 ts_tablet_manager.cc:1367] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 08:14:35.728128 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:35.728355 32133 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:35.728263 1154 raft_consensus.cc:348] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.728487 1154 raft_consensus.cc:374] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 08:14:35.728616 1154 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Initialized, Role: FOLLOWER
I20240501 08:14:35.728664 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:35.728894 31872 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:35.728958 1154 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.729815 1154 ts_tablet_manager.cc:1398] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 08:14:35.730656 32133 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:35.731113 31872 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 1.
I20240501 08:14:35.731542 31939 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:35.731758 1146 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:35.731917 1166 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:35.732024 1166 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:35.732223 1166 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:35.734998 31727 catalog_manager.cc:5484] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 1 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:35.847671 1082 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:35.847846 1082 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.848515 1082 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:35.848976 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782" is_pre_election: true
I20240501 08:14:35.849045 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:35.849241 32133 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:35.849285 32003 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 0.
I20240501 08:14:35.849637 32198 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:35.849889 1082 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:35.850013 1082 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:35.850106 1082 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:35.852041 1082 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.852610 1082 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:35.852995 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:35.853118 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:35.853240 32133 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:35.853374 32003 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:35.855530 32133 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:35.855649 32003 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 1.
I20240501 08:14:35.855902 32198 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:35.856192 1082 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:35.856439 1082 raft_consensus.cc:686] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:35.856731 1082 consensus_queue.cc:237] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.859694 31727 catalog_manager.cc:5484] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 1 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:35.981770 1160 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 08:14:35.982005 1160 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.982654 1160 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:35.983246 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4" is_pre_election: true
I20240501 08:14:35.983266 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf" is_pre_election: true
I20240501 08:14:35.983446 32263 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:35.983484 32003 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 0.
I20240501 08:14:35.983928 32068 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:35.984205 1160 raft_consensus.cc:2781] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 08:14:35.984340 1160 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 08:14:35.984424 1160 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:35.986444 1160 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.986994 1160 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:35.987586 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:35.987644 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:35.987785 32003 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:35.987829 32263 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 08:14:35.989794 32263 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:35.989863 32003 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 1.
I20240501 08:14:35.990255 32068 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:35.990540 1160 raft_consensus.cc:2781] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 08:14:35.990679 1160 raft_consensus.cc:686] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:35.990993 1160 consensus_queue.cc:237] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:35.994189 31727 catalog_manager.cc:5484] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 1 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:36.279696 1166 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:36.285631 1173 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 08:14:36.394647 1170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:36.395004 1170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:36.403432 1170 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 08:14:36.408183 1082 consensus_queue.cc:1035] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:36.416446 1082 consensus_queue.cc:1035] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:36.433977 1160 consensus_queue.cc:1035] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 08:14:36.442077 1160 consensus_queue.cc:1035] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 247.631 ms
time per row: 0.123815 ms
-----------------------------------------------------
Suppressions used:
count bytes template
28 7248 libcrypto.so.1.1
-----------------------------------------------------
I20240501 08:14:37.788563 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.69:34467
--local_ip_for_outbound_sockets=127.6.56.69
--tserver_master_addrs=127.6.56.126:36653
--webserver_port=44197
--webserver_interface=127.6.56.69
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240501 08:14:38.140501 1241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:38.140833 1241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:38.140990 1241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:38.149899 1241 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:38.150121 1241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:38.150326 1241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.69
I20240501 08:14:38.160079 1241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.69:34467
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.56.69
--webserver_port=44197
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.69
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:38.161692 1241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:38.163460 1241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:38.177018 1248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:38.177716 1250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:38.178380 1247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:38.179265 1241 server_base.cc:1034] running on GCE node
I20240501 08:14:38.179880 1241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:38.180941 1241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:38.182210 1241 hybrid_clock.cc:648] HybridClock initialized: now 1714551278182141 us; error 58 us; skew 500 ppm
I20240501 08:14:38.182627 1241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:38.185087 1241 webserver.cc:460] Webserver started at http://127.6.56.69:44197/ using document root <none> and password file <none>
I20240501 08:14:38.185897 1241 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:38.186020 1241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:38.191943 1241 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:14:38.194813 1256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:38.196535 1241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240501 08:14:38.196707 1241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b9c992291081480abb54b0415c9385f0"
format_stamp: "Formatted at 2024-05-01 08:14:16 on dist-test-slave-f1nk"
I20240501 08:14:38.197497 1241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:38.248404 1241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:38.249235 1241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:38.249517 1241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:38.250422 1241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:38.251971 1241 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:38.252045 1241 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:38.252138 1241 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:38.252182 1241 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:38.284507 1241 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.69:34467
I20240501 08:14:38.284520 1368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.69:34467 every 8 connection(s)
I20240501 08:14:38.286322 1241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 08:14:38.292694 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 1241
I20240501 08:14:38.293332 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.70:34611
--local_ip_for_outbound_sockets=127.6.56.70
--tserver_master_addrs=127.6.56.126:36653
--webserver_port=38207
--webserver_interface=127.6.56.70
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:38.299849 1369 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:38.300176 1369 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:38.300926 1369 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:38.302524 31727 ts_manager.cc:194] Re-registered known tserver with Master: b9c992291081480abb54b0415c9385f0 (127.6.56.69:34467)
I20240501 08:14:38.303753 31727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.69:37039
W20240501 08:14:38.647121 1373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:38.647461 1373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:38.647626 1373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:38.656157 1373 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:38.656338 1373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:38.656502 1373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.70
I20240501 08:14:38.665635 1373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.70:34611
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.56.70
--webserver_port=38207
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.70
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:38.667011 1373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:38.668771 1373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:38.680999 1382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:38.681648 1380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:38.682348 1379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:38.683270 1373 server_base.cc:1034] running on GCE node
I20240501 08:14:38.683871 1373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:38.684839 1373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:38.686038 1373 hybrid_clock.cc:648] HybridClock initialized: now 1714551278685997 us; error 37 us; skew 500 ppm
I20240501 08:14:38.686347 1373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:38.688771 1373 webserver.cc:460] Webserver started at http://127.6.56.70:38207/ using document root <none> and password file <none>
I20240501 08:14:38.689491 1373 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:38.689607 1373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:38.694890 1373 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 08:14:38.697554 1388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:38.698869 1373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:14:38.699019 1373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "a86d83e684ff407fac40b08fa600a85b"
format_stamp: "Formatted at 2024-05-01 08:14:16 on dist-test-slave-f1nk"
I20240501 08:14:38.699779 1373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:38.733981 1373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:38.734827 1373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:38.735116 1373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:38.736171 1373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:38.738006 1373 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:38.738103 1373 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:38.738234 1373 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:38.738346 1373 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240501 08:14:38.769559 1373 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.70:34611
I20240501 08:14:38.769623 1500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.70:34611 every 8 connection(s)
I20240501 08:14:38.771005 1373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 08:14:38.779402 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 1373
I20240501 08:14:38.780144 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.71:44621
--local_ip_for_outbound_sockets=127.6.56.71
--tserver_master_addrs=127.6.56.126:36653
--webserver_port=45545
--webserver_interface=127.6.56.71
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:38.784165 1501 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:38.784497 1501 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:38.785264 1501 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:38.787129 31727 ts_manager.cc:194] Re-registered known tserver with Master: a86d83e684ff407fac40b08fa600a85b (127.6.56.70:34611)
I20240501 08:14:38.787910 31727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.70:38961
W20240501 08:14:39.144361 1504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:39.144685 1504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:39.144783 1504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:39.153640 1504 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:39.153826 1504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:39.154026 1504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.71
I20240501 08:14:39.163792 1504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.71:44621
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.56.71
--webserver_port=45545
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.71
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:39.165210 1504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:39.167135 1504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:39.180191 1512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:39.180653 1509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:39.181311 1510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:39.182137 1504 server_base.cc:1034] running on GCE node
I20240501 08:14:39.182868 1504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:39.183952 1504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:39.185201 1504 hybrid_clock.cc:648] HybridClock initialized: now 1714551279185144 us; error 36 us; skew 500 ppm
I20240501 08:14:39.185534 1504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:39.187976 1504 webserver.cc:460] Webserver started at http://127.6.56.71:45545/ using document root <none> and password file <none>
I20240501 08:14:39.188777 1504 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:39.188917 1504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:39.194321 1504 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:14:39.197389 1518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:39.198747 1504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 08:14:39.198907 1504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f065d94513dc485293fd2e673e08918a"
format_stamp: "Formatted at 2024-05-01 08:14:17 on dist-test-slave-f1nk"
I20240501 08:14:39.199710 1504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:39.219158 1504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:39.220013 1504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:39.220328 1504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:39.221276 1504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:39.222968 1504 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:39.223052 1504 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:39.223183 1504 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:39.223239 1504 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:39.255380 1504 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.71:44621
I20240501 08:14:39.255393 1630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.71:44621 every 8 connection(s)
I20240501 08:14:39.256989 1504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 08:14:39.266337 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 1504
I20240501 08:14:39.266961 6369 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
/tmp/dist-test-taskFwDPgb/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.56.72:42943
--local_ip_for_outbound_sockets=127.6.56.72
--tserver_master_addrs=127.6.56.126:36653
--webserver_port=45979
--webserver_interface=127.6.56.72
--builtin_ntp_servers=127.6.56.84:42721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240501 08:14:39.269769 1631 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:39.270097 1631 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:39.271010 1631 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:39.274528 31727 ts_manager.cc:194] Re-registered known tserver with Master: f065d94513dc485293fd2e673e08918a (127.6.56.71:44621)
I20240501 08:14:39.275439 31727 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.71:57293
I20240501 08:14:39.306453 1369 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
W20240501 08:14:39.629230 1634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 08:14:39.629550 1634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 08:14:39.629699 1634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 08:14:39.637362 1634 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240501 08:14:39.637529 1634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 08:14:39.637698 1634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.56.72
I20240501 08:14:39.648576 1634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.56.84:42721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.56.72:42943
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.56.72
--webserver_port=45979
--tserver_master_addrs=127.6.56.126:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.56.72
--log_dir=/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 07:43:24 UTC on efd1337a2aa3
build id 71
ASAN enabled
I20240501 08:14:39.650122 1634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 08:14:39.652172 1634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 08:14:39.664813 1639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:39.665374 1640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 08:14:39.666168 1642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 08:14:39.666873 1634 server_base.cc:1034] running on GCE node
I20240501 08:14:39.667699 1634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 08:14:39.668797 1634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 08:14:39.670025 1634 hybrid_clock.cc:648] HybridClock initialized: now 1714551279669961 us; error 59 us; skew 500 ppm
I20240501 08:14:39.670356 1634 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 08:14:39.672895 1634 webserver.cc:460] Webserver started at http://127.6.56.72:45979/ using document root <none> and password file <none>
I20240501 08:14:39.673681 1634 fs_manager.cc:359] Metadata directory not provided
I20240501 08:14:39.673823 1634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 08:14:39.679358 1634 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 08:14:39.682221 1648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:39.683652 1634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 08:14:39.683844 1634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e4e49c68247b499297c4fbeaf145e11e"
format_stamp: "Formatted at 2024-05-01 08:14:17 on dist-test-slave-f1nk"
I20240501 08:14:39.684778 1634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 08:14:39.700726 1634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 08:14:39.701555 1634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 08:14:39.701836 1634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 08:14:39.702796 1634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 08:14:39.704325 1634 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 08:14:39.704412 1634 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:39.704545 1634 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 08:14:39.704598 1634 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 08:14:39.742516 1634 rpc_server.cc:307] RPC server started. Bound to: 127.6.56.72:42943
I20240501 08:14:39.742553 1760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.56.72:42943 every 8 connection(s)
I20240501 08:14:39.744004 1634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFwDPgb/test-tmp/rebalancer_tool-test.0.RebalancingDuringElectionStormTest.RoundRobin_0.1714551039964266-6369-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 08:14:39.751314 6369 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFwDPgb/build/asan/bin/kudu as pid 1634
I20240501 08:14:39.758837 1761 heartbeater.cc:347] Connected to a master server at 127.6.56.126:36653
I20240501 08:14:39.759130 1761 heartbeater.cc:464] Registering TS with master...
I20240501 08:14:39.760076 1761 heartbeater.cc:511] Master 127.6.56.126:36653 requested a full tablet report, sending...
I20240501 08:14:39.761612 31731 ts_manager.cc:194] Re-registered known tserver with Master: e4e49c68247b499297c4fbeaf145e11e (127.6.56.72:42943)
I20240501 08:14:39.762723 31731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.56.72:49251
I20240501 08:14:39.783715 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.783982 32002 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.784142 32002 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.786208 32002 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.787117 32002 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:39.787544 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.787665 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.787771 32132 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.787883 32262 raft_consensus.cc:3032] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.787984 32262 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:39.788342 32262 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.788519 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.788829 32002 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.788933 32002 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.789016 32262 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.790413 32132 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.791049 1501 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
I20240501 08:14:39.791754 32262 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.791785 32002 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:39.792222 31938 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.792558 1239 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.793673 1239 raft_consensus.cc:686] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.793947 1239 consensus_queue.cc:237] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.795601 32002 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:39.795933 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.796216 32132 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.796272 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.796502 31871 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.796607 31871 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:39.796763 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.796931 32003 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.796882 31871 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:39.797021 32003 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.797407 31871 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.799043 32132 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.799544 32003 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.800009 32003 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:39.800032 31871 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.800563 31938 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.800958 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.801188 31871 raft_consensus.cc:3032] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.801213 1239 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.801348 31871 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:39.801376 1239 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.801649 1239 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:39.801663 31731 catalog_manager.cc:5484] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:39.801749 31871 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.802261 31871 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.803002 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.803238 32262 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.803741 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.803925 32003 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.804100 32003 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.805328 31871 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.805773 31939 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:39.806097 1769 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.806280 32262 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.807214 32003 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.808913 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.809327 32132 raft_consensus.cc:3032] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.809691 32132 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:39.810576 32132 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.811800 32003 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:39.811861 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.812083 32262 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.812964 32001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.813153 32001 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.813268 32001 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.813798 1769 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.814205 1769 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.814949 32262 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.815380 31938 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.815763 1773 raft_consensus.cc:2781] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.816267 1773 raft_consensus.cc:686] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.816152 32001 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.816691 1773 consensus_queue.cc:237] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.818096 32001 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:39.818883 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.819104 31871 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.821996 31871 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.822484 31939 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:39.822835 1239 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.824780 32001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.825088 32001 raft_consensus.cc:465] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:14:39.826170 32001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.826531 32001 raft_consensus.cc:465] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:14:39.827572 32001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.827858 32001 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.828039 32001 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.823652 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.830849 32133 raft_consensus.cc:3032] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.830984 32133 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:39.831292 32133 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.831949 32133 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.835510 32133 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.837255 32132 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.841022 32132 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.843550 1239 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.843948 1239 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.843740 32001 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.846030 32001 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:39.846791 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.847018 32262 raft_consensus.cc:3032] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.846905 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.847119 32262 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:39.847198 31871 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.847400 32262 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.847754 32262 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.850620 32262 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.851078 31871 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.851579 31938 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.851909 1773 raft_consensus.cc:2781] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.852070 1773 raft_consensus.cc:686] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.852344 1773 consensus_queue.cc:237] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.853729 32001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.853941 32001 raft_consensus.cc:465] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:14:39.855089 32001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.855382 32001 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.855620 32001 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.861155 32001 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:39.862043 32001 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:39.863240 32001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.863437 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.863534 32001 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.863662 32001 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.863668 32262 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.867394 32001 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.868315 32001 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:39.868784 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.868840 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.868984 31871 raft_consensus.cc:3032] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.869081 32133 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.869092 31871 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:39.869342 31871 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.869652 31871 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.871513 31871 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.871721 32133 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.872043 31939 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:39.872565 1239 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.872735 1239 raft_consensus.cc:686] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.872812 32001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f78273f47414fd199e593595e6aae23"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.873337 32262 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.873049 1239 consensus_queue.cc:237] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.873817 31938 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.874130 1773 raft_consensus.cc:2781] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.874269 1773 raft_consensus.cc:686] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.874532 1773 consensus_queue.cc:237] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:39.873044 32001 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.874969 32001 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.876909 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.877121 32132 raft_consensus.cc:3032] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.877224 32132 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:39.877470 32132 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:39.877840 32132 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.877667 32001 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.878775 32001 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:39.879657 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.879678 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.879848 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.879930 32262 raft_consensus.cc:3032] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.880019 32262 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:39.880038 32002 raft_consensus.cc:465] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:14:39.880048 32133 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.880354 32262 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.880744 32262 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.881017 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.881189 32002 raft_consensus.cc:465] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:14:39.881968 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e872bea48fc479686691121187aa0db"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.882124 32002 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.882242 32002 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.883811 32262 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.883893 32133 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.884253 31938 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.884570 1773 raft_consensus.cc:2781] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.884737 1773 raft_consensus.cc:686] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.884879 32002 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.885005 1773 consensus_queue.cc:237] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.885445 32002 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:39.885941 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.886150 32262 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.886286 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.886497 32003 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.886605 32003 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.887164 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.887650 32133 raft_consensus.cc:3032] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.887753 32133 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:39.889078 32262 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.889503 31938 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.889843 1777 raft_consensus.cc:2781] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.889997 1777 raft_consensus.cc:686] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.890185 32133 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.890314 1777 consensus_queue.cc:237] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.890573 32133 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.889941 32003 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.893232 32003 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:39.893774 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.893872 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.894026 31871 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.894090 32263 raft_consensus.cc:3032] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.894173 32263 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:39.894543 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.894451 32263 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.894701 32003 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.894805 32003 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.894837 32263 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.896898 31871 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.897305 31939 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:39.897568 32263 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.897627 1239 raft_consensus.cc:2781] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.897537 32003 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.897765 1239 raft_consensus.cc:686] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.898084 1239 consensus_queue.cc:237] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.898241 32003 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:39.899130 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.899176 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93a196489f264058a064dc84690ba947"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.899395 32131 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.899411 32000 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.899541 32000 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.899588 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.899793 31872 raft_consensus.cc:3032] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.899896 31872 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:39.900141 31872 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.900552 31872 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.902534 32000 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:39.903062 32131 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.903198 32000 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:39.903211 31872 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.903666 31939 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:39.903973 1776 raft_consensus.cc:2781] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.904110 1776 raft_consensus.cc:686] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.904248 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.904433 31872 raft_consensus.cc:3032] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.904529 31872 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:39.904467 1776 consensus_queue.cc:237] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.905154 31872 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:39.905602 31872 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.907348 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.907483 32132 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.907608 32131 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.908610 31872 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.909149 31939 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:39.909561 1777 raft_consensus.cc:2781] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.909821 1777 raft_consensus.cc:686] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.910071 1777 consensus_queue.cc:237] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:39.910358 32131 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.910632 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.910861 32000 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.910975 32000 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.914438 32000 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.915882 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.916198 32262 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.917762 32000 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:39.918304 32133 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.918475 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.918771 31872 raft_consensus.cc:3032] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.918782 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.918984 32003 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.919147 32003 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.920567 31872 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:39.920944 31872 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.921335 31872 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.922782 32003 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.924170 32003 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:39.925006 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.925079 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.925253 32263 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.925290 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.925467 32000 raft_consensus.cc:465] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:14:39.925467 32133 raft_consensus.cc:3032] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.925652 32133 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:39.925995 32133 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.926276 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.926502 32133 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.926470 32002 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.926795 32002 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.926466 31872 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.927345 31939 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:39.929184 1239 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.929356 1239 raft_consensus.cc:686] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.929738 1239 consensus_queue.cc:237] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.929841 32002 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.931460 32002 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:39.931597 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.931784 32263 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.931857 32261 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.932027 32133 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.932252 32262 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.932480 31938 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.932627 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:39.932811 32002 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.932937 32002 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.931883 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:39.933130 32131 raft_consensus.cc:3032] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.933260 32131 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:39.933619 32131 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.934108 32131 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.934727 32261 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.935143 31938 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.935529 1776 raft_consensus.cc:2781] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.935750 1776 raft_consensus.cc:686] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.936082 1776 consensus_queue.cc:237] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.935755 32002 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.937310 32002 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:39.937359 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.937443 1769 raft_consensus.cc:2781] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.937570 32261 raft_consensus.cc:3032] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.937664 32261 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:39.937981 32261 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.938472 32261 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.941929 32261 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.942519 31938 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:39.947137 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.947469 31872 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.950232 1239 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:39.950393 1239 raft_consensus.cc:686] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.950706 1239 consensus_queue.cc:237] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:39.951268 31872 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.956140 32131 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 2.
I20240501 08:14:39.969337 1769 raft_consensus.cc:686] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:39.969686 1769 consensus_queue.cc:237] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:39.975440 31731 catalog_manager.cc:5484] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:39.976191 31731 catalog_manager.cc:5484] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:39.976694 31731 catalog_manager.cc:5484] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:39.977038 31731 catalog_manager.cc:5484] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:39.977365 31731 catalog_manager.cc:5484] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:39.977646 31731 catalog_manager.cc:5484] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:39.977968 31731 catalog_manager.cc:5484] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:39.978286 31731 catalog_manager.cc:5484] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:39.978582 31731 catalog_manager.cc:5484] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:39.978899 31731 catalog_manager.cc:5484] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:39.979197 31731 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:39.979482 31731 catalog_manager.cc:5484] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:39.979820 31731 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:39.980221 31731 catalog_manager.cc:5484] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:39.980659 31731 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:39.981050 31731 catalog_manager.cc:5484] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 1 to 2, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 2 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:39.990684 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:39.991024 32131 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:39.991232 32131 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.994515 32131 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.995702 32131 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:39.996205 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:39.996204 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:39.996482 31872 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.996482 32261 raft_consensus.cc:3032] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:39.996575 32261 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:39.996908 32261 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:39.997267 32261 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:39.999348 31872 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:39.999907 32261 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:39.999962 32069 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.000487 1245 raft_consensus.cc:2781] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.001084 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.001380 32131 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.001621 32131 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.005920 32131 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.007680 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.007942 32001 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.008059 32001 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.007778 32131 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.008481 32001 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.008992 32001 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.009554 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.009860 32133 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.010032 32133 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.010283 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.010473 31872 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.012336 31872 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.012794 32069 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.013156 1799 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.013078 32133 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.013396 1799 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.013778 1799 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.014350 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.014565 32003 raft_consensus.cc:3032] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.014663 32003 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.014917 32003 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.015147 32001 raft_consensus.cc:2396] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.015254 32003 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.015936 32133 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.016609 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.016858 32261 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.017028 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93a196489f264058a064dc84690ba947"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.017604 32131 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.017720 32131 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.018604 32003 raft_consensus.cc:2396] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.019658 32261 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.020256 32068 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:40.020607 32131 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.021426 32131 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.022624 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.022900 32133 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.023062 32133 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.023344 1798 raft_consensus.cc:2781] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.023528 1245 raft_consensus.cc:686] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.023972 1245 consensus_queue.cc:237] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.024516 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.024786 31872 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.027334 32133 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.027993 32133 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.028417 1798 raft_consensus.cc:686] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.028982 31872 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.029107 1798 consensus_queue.cc:237] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.031477 32069 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.031754 1798 raft_consensus.cc:2781] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.031898 1798 raft_consensus.cc:686] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.031908 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.032141 1798 consensus_queue.cc:237] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.032248 31872 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.035970 31872 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.036685 32069 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.037129 1245 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.037354 1245 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.037704 1245 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.037990 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.036705 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.038192 32003 raft_consensus.cc:3032] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.038280 32131 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.038295 32003 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.038378 32131 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.038550 32003 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.038899 32003 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.041326 32131 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.042090 32131 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.042500 32003 raft_consensus.cc:2396] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.043849 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.044100 32003 raft_consensus.cc:3032] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.044210 32003 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.044687 32003 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.045189 32003 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.045317 31731 catalog_manager.cc:5484] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 1 to 2, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.048460 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.048707 32261 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.051143 32003 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.051465 32261 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.051606 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.051920 32131 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.052098 32131 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.052848 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.053081 32001 raft_consensus.cc:3032] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.053174 32001 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.053498 32001 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.053931 32001 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.055859 32131 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.057008 32001 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.057020 32131 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.058121 32068 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:40.059013 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.059489 32133 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.059619 32133 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.062436 32133 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.063138 32133 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.063678 1798 raft_consensus.cc:2781] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.063829 1798 raft_consensus.cc:686] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.064121 1798 consensus_queue.cc:237] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.064204 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.064563 32001 raft_consensus.cc:3032] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.064718 32001 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.065059 32001 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.065475 32001 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.069106 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.069495 32133 raft_consensus.cc:465] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:14:40.070222 32001 raft_consensus.cc:2396] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.070715 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.070870 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.070981 32133 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.071084 32133 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.071108 32261 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.074285 32261 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.074908 32133 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.075644 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.075838 32003 raft_consensus.cc:3032] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.075924 32003 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.076169 32003 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.076311 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.076483 32003 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.076599 31872 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.079295 32003 raft_consensus.cc:2396] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.080152 31872 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.080250 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.080509 32261 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.082739 32069 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.083261 1801 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.083541 32261 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.084316 32068 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:40.084811 1799 raft_consensus.cc:2781] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.084944 1799 raft_consensus.cc:686] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.085187 1799 consensus_queue.cc:237] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.093693 32133 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.096174 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.101440 32133 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.101994 32133 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.112726 1801 raft_consensus.cc:686] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.112833 32068 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:40.113181 1801 consensus_queue.cc:237] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.113823 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.114149 32001 raft_consensus.cc:3032] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.114267 32001 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.114617 32001 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.115015 32001 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.116222 32133 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.116914 32133 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.117874 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.118101 31872 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.118407 32001 raft_consensus.cc:2396] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.120080 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.120369 32001 raft_consensus.cc:3032] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.120462 32001 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.120777 32001 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.120848 31872 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.120671 31731 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:40.121052 32001 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.121213 31731 catalog_manager.cc:5484] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.121649 31731 catalog_manager.cc:5484] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:40.122021 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.122231 32131 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.122088 31731 catalog_manager.cc:5484] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 1 to 2, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.122340 32131 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.122488 31731 catalog_manager.cc:5484] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 reported cstate change: term changed from 1 to 2, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.123989 32001 raft_consensus.cc:2396] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.124567 32069 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.124761 1807 raft_consensus.cc:2781] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.124914 1807 raft_consensus.cc:686] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.125072 1808 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.125211 1808 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.125276 1807 consensus_queue.cc:237] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.125530 1808 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.127815 32131 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.129137 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.129189 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.129406 31872 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.129410 32261 raft_consensus.cc:3032] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.129503 32261 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.129783 32261 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.130111 32261 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.130138 32133 raft_consensus.cc:1226] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Rejecting Update request from peer 64f7786244bd4f90af1ad476966482c4 for earlier term 1. Current term is 2. Ops: []
I20240501 08:14:40.129709 32131 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.132009 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f78273f47414fd199e593595e6aae23"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.132174 32131 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.132351 32131 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.132544 31872 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.132972 32069 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.133005 32261 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.133396 1245 raft_consensus.cc:2781] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.133592 1245 raft_consensus.cc:686] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.133966 1245 consensus_queue.cc:237] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.135375 32131 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.136090 32131 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.137109 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.137300 32131 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.137300 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.137414 32131 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.137526 32001 raft_consensus.cc:3032] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.137610 32001 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.137923 32001 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.138196 32001 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.141227 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.141449 32263 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.141995 32131 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.142709 32001 raft_consensus.cc:2396] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.143054 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.143263 31872 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.144173 32263 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.144855 32068 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.145164 1801 raft_consensus.cc:2781] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.145345 1801 raft_consensus.cc:686] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.145706 1801 consensus_queue.cc:237] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.147049 31872 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.147617 32069 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.148015 1799 raft_consensus.cc:2781] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.148172 1799 raft_consensus.cc:686] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.147956 31731 catalog_manager.cc:5484] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.148514 1799 consensus_queue.cc:237] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.148451 31731 catalog_manager.cc:5484] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:40.148886 31731 catalog_manager.cc:5484] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.149246 31731 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.154153 32131 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.155543 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.155790 32133 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.155903 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.156014 32133 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.156140 32263 raft_consensus.cc:3032] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.156215 32263 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.156543 32263 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.156996 32263 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.159754 32133 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.160405 32263 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.161293 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.161358 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.161556 32263 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.161561 31872 raft_consensus.cc:3032] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.161649 31872 raft_consensus.cc:729] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.161990 31872 consensus_queue.cc:260] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.162437 31872 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.164166 32263 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.164831 32068 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:40.165161 1807 raft_consensus.cc:2781] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.165416 1807 raft_consensus.cc:686] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.165680 1807 consensus_queue.cc:237] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.165904 31872 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.166388 32133 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.168684 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.168951 32131 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.169111 32131 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.173617 32131 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.174952 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.175222 31872 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.177959 31872 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.178444 32069 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.178783 1801 raft_consensus.cc:2781] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.178975 1801 raft_consensus.cc:686] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.179428 32131 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.179394 1801 consensus_queue.cc:237] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.180202 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.180455 32001 raft_consensus.cc:3032] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.180544 32001 raft_consensus.cc:729] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.180862 32001 consensus_queue.cc:260] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.181238 32001 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.183280 31731 catalog_manager.cc:5484] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 reported cstate change: term changed from 1 to 2, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.183491 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e872bea48fc479686691121187aa0db"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.183694 32131 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.183786 32131 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.183771 31731 catalog_manager.cc:5484] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.184183 31731 catalog_manager.cc:5484] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 reported cstate change: term changed from 1 to 2, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.184571 32001 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.189145 32131 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.191097 31731 catalog_manager.cc:5484] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 1 to 2, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.191674 31731 catalog_manager.cc:5484] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 reported cstate change: term changed from 1 to 2, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.191848 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.192063 32001 raft_consensus.cc:3032] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.192159 32001 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.192437 32001 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.192725 32001 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.196156 32131 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.196728 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.196935 32263 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.198055 32001 raft_consensus.cc:2396] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.198437 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.198683 32131 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.198868 32131 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.199743 32263 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.200528 32068 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.200850 1801 raft_consensus.cc:2781] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.201006 1801 raft_consensus.cc:686] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.201282 1801 consensus_queue.cc:237] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.202193 32131 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.203305 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.203511 32001 raft_consensus.cc:3032] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.203612 32001 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.203934 32001 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.204242 32001 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.207803 32001 raft_consensus.cc:2396] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.209100 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.209365 32263 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.213701 32263 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.213676 31731 catalog_manager.cc:5484] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.215723 32131 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.216853 32068 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.217360 1801 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.217613 1801 raft_consensus.cc:686] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.218298 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.218526 32131 raft_consensus.cc:465] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240501 08:14:40.218000 1801 consensus_queue.cc:237] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.219471 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.219691 32131 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.219843 32131 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.223021 32131 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.224400 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.224671 31872 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.228132 31872 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.228843 32069 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.229218 1245 raft_consensus.cc:2781] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.229395 1245 raft_consensus.cc:686] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.229722 1245 consensus_queue.cc:237] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.231235 32131 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.231751 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.231993 32001 raft_consensus.cc:3032] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.232074 32001 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.232386 32001 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.232528 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.232714 32001 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.232895 32131 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.233080 32131 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.235705 32001 raft_consensus.cc:2396] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.237267 32131 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.238754 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.239153 31872 raft_consensus.cc:3032] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.239320 31872 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.239820 31872 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.240492 31872 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.244433 31872 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.245754 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.246032 32263 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.246150 32131 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.246987 32069 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.247471 1801 raft_consensus.cc:2781] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.247365 31731 catalog_manager.cc:5484] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.247669 1801 raft_consensus.cc:686] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.248034 1801 consensus_queue.cc:237] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.248912 32263 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.250929 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.251330 32131 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.251539 32131 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.254638 31731 catalog_manager.cc:5484] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.260527 32131 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.261255 32131 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.261773 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.262030 32001 raft_consensus.cc:3032] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.262117 32001 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.262042 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.262259 32263 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.262454 32001 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.262804 32001 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.262984 31731 catalog_manager.cc:5484] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 1 to 2, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.265098 32263 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 3.
I20240501 08:14:40.265512 32068 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:40.265869 1245 raft_consensus.cc:2781] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.266005 1245 raft_consensus.cc:686] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.266305 1245 consensus_queue.cc:237] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.267670 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:40.267910 32131 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.267974 32001 raft_consensus.cc:2396] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.268150 32131 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.271152 32131 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.272404 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.272651 31872 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.275511 31872 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.276121 32069 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.276475 1801 raft_consensus.cc:2781] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.276646 1801 raft_consensus.cc:686] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.276957 1801 consensus_queue.cc:237] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.277321 32131 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.278155 1631 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
I20240501 08:14:40.278318 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.278545 32001 raft_consensus.cc:3032] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.278620 32001 raft_consensus.cc:729] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.278894 32001 consensus_queue.cc:260] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.279254 32001 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.281848 31731 catalog_manager.cc:5484] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 3 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.282518 32001 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 2.
I20240501 08:14:40.294117 31731 catalog_manager.cc:5484] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 1 to 2, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 2 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:40.310353 32263 raft_consensus.cc:1261] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.311064 1769 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.321185 31872 raft_consensus.cc:1261] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.322373 1776 consensus_queue.cc:1035] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.322353 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.323637 32262 raft_consensus.cc:1261] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.325663 1777 consensus_queue.cc:1035] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.334632 31871 raft_consensus.cc:1261] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.335520 1776 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.353299 32261 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.353518 32261 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.354432 31871 raft_consensus.cc:1261] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.355196 1776 consensus_queue.cc:1035] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.358032 32261 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.360607 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.360997 31871 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.365659 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.365988 32001 raft_consensus.cc:3032] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.366088 32001 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.366451 32001 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.366866 32001 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.370774 32262 raft_consensus.cc:1261] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.371399 1777 consensus_queue.cc:1035] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.371457 32001 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.371799 32261 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.372162 32198 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:40.372542 1240 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.375015 1240 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.375522 1240 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.377689 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.378046 32261 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.378219 32261 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.380861 31871 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.383358 31731 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.384964 32261 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.385811 32261 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.386294 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.386649 31871 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.387765 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.388015 32131 raft_consensus.cc:3032] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.388110 32131 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.389211 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.389453 32261 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.389591 32261 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.389716 32131 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.390033 32131 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.392668 32261 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.392951 32131 raft_consensus.cc:2396] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.393445 32261 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.394102 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.394381 32001 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.394552 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.394650 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.394804 32131 raft_consensus.cc:3032] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.394897 32131 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.395020 32261 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.395185 32261 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.395184 32131 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.395466 32131 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.397399 32001 raft_consensus.cc:2396] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.398332 32131 raft_consensus.cc:2396] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.398234 32261 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.398751 32198 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.399111 32261 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.399652 1240 raft_consensus.cc:2726] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.399892 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.400090 32131 raft_consensus.cc:3032] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.399871 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.400171 32131 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.400308 32001 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.400307 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.400591 32261 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.400751 32261 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.400477 32131 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.401664 32131 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.402644 31871 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.403070 32001 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.403174 32199 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:40.403478 32198 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:40.403750 1240 raft_consensus.cc:2781] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.403862 1839 raft_consensus.cc:2781] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.403911 1240 raft_consensus.cc:686] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.404067 32261 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.404232 1240 consensus_queue.cc:237] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.405391 32261 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.405673 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.406121 31870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.406049 32001 raft_consensus.cc:3032] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.406304 31870 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.406329 32001 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.406633 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.406877 32261 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.407024 32261 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.407843 32001 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.408433 32001 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.409197 31870 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.410059 32199 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.410424 1840 raft_consensus.cc:2781] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.410552 1840 raft_consensus.cc:686] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.410879 1839 raft_consensus.cc:686] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.410931 1840 consensus_queue.cc:237] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.411247 1839 consensus_queue.cc:237] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.415297 32001 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.416338 32261 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.418628 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.418993 32261 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.419263 32001 raft_consensus.cc:3032] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.419776 32001 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.419915 32131 raft_consensus.cc:2396] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.420524 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.420701 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.420986 32001 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.421402 31871 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.421440 32001 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.422150 32261 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.422323 32261 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.425840 32261 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.425504 31871 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 2.
I20240501 08:14:40.427793 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.428025 32133 raft_consensus.cc:3032] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.428107 32133 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.428077 32199 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.428452 1240 raft_consensus.cc:2781] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.427363 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.428642 32003 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.428720 1240 raft_consensus.cc:686] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.429064 1240 consensus_queue.cc:237] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.429275 31727 catalog_manager.cc:5484] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.429787 31727 catalog_manager.cc:5484] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.431368 32003 raft_consensus.cc:2396] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.432335 32003 raft_consensus.cc:1261] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 08:14:40.432170 32133 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.432554 32133 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.432598 32003 pending_rounds.cc:76] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 3
I20240501 08:14:40.432752 32003 pending_rounds.cc:97] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:40.432977 32003 raft_consensus.cc:2866] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:40.434705 32133 raft_consensus.cc:2396] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.435192 32198 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.435632 1240 raft_consensus.cc:2726] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.436401 1801 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.442685 32001 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 2.
I20240501 08:14:40.443120 31727 catalog_manager.cc:5484] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 1 to 2, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 2 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.443946 31727 catalog_manager.cc:5484] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.426513 32261 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.447795 31871 raft_consensus.cc:1261] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 08:14:40.448489 1245 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.449211 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f78273f47414fd199e593595e6aae23"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.449508 32261 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.449872 32261 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.452878 32261 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.453531 32261 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.454077 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.454320 32001 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.454388 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.454602 32133 raft_consensus.cc:3032] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.454697 32133 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.454988 32133 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.455281 32133 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.457366 32001 raft_consensus.cc:2396] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.459420 32261 raft_consensus.cc:1261] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.459750 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.459966 32263 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.460080 32263 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.460111 1777 consensus_queue.cc:1035] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.460664 31871 raft_consensus.cc:1261] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.461169 1777 consensus_queue.cc:1035] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.462877 32263 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.463757 32263 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 2 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.464174 31870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.464412 31870 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.464774 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.465029 32001 raft_consensus.cc:3032] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.465173 32001 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.465190 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.465457 32263 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.465621 32263 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.465492 32001 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.465965 32001 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.467283 31870 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 2.
I20240501 08:14:40.467806 32199 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.468091 1240 raft_consensus.cc:2781] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.468254 1240 raft_consensus.cc:686] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.468432 32263 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.468557 1240 consensus_queue.cc:237] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.469106 32263 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.470013 32262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.470075 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.470222 32262 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.470317 32262 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.470341 31872 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.471460 32133 raft_consensus.cc:2396] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.471932 32198 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.472510 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.472775 32131 raft_consensus.cc:3032] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.472867 32131 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.472956 32262 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.473120 32001 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 2.
I20240501 08:14:40.473198 32131 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.473556 32262 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.473558 32131 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.474064 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.474265 32003 raft_consensus.cc:3032] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.474342 32003 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.474364 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.474558 31869 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.474598 32003 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.474954 32003 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.476493 1840 raft_consensus.cc:2726] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.477536 31869 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.477672 32003 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.478029 32199 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.478406 1840 raft_consensus.cc:2781] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.478546 1840 raft_consensus.cc:686] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.478936 31872 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.478881 1840 consensus_queue.cc:237] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.479539 32199 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.480041 1839 raft_consensus.cc:2781] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.480238 1839 raft_consensus.cc:686] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.480564 1839 consensus_queue.cc:237] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.481349 32262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.481618 32262 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.482410 32262 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.483345 32131 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.487731 32262 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.489796 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.490103 31872 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.490664 31870 raft_consensus.cc:1261] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.491062 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.491473 1777 consensus_queue.cc:1035] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.491899 32131 raft_consensus.cc:3032] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.492235 32131 raft_consensus.cc:729] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.495368 32131 consensus_queue.cc:260] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.495862 32131 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.495931 31727 catalog_manager.cc:5484] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.496718 31727 catalog_manager.cc:5484] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 1 to 2, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 2 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.497342 31727 catalog_manager.cc:5484] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.500227 31872 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.500934 32199 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.501741 32262 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.502003 1240 raft_consensus.cc:2781] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.503796 32131 raft_consensus.cc:2396] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.504101 1240 raft_consensus.cc:686] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.504492 32262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e872bea48fc479686691121187aa0db"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.505363 32262 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.505522 32262 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.504945 1240 consensus_queue.cc:237] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.510437 32262 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.511286 32262 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.511685 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.511881 32001 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.512352 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.512647 32133 raft_consensus.cc:3032] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.512778 32133 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.513125 32133 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.513484 32133 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.514896 32001 raft_consensus.cc:2396] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.515753 32262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.516003 32262 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.516166 32262 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.519495 32262 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.519872 32261 raft_consensus.cc:1226] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Rejecting Update request from peer 6b7faa87de354980a76c80fb864f9edf for earlier term 2. Current term is 3. Ops: []
I20240501 08:14:40.520478 1776 consensus_queue.cc:1046] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.520601 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.520825 1776 raft_consensus.cc:3032] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.520848 31869 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.520931 1776 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.521229 1776 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.521593 1776 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.522348 32262 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.523830 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.524187 31869 raft_consensus.cc:2396] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.525039 32133 raft_consensus.cc:2396] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.525548 32001 raft_consensus.cc:2396] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.525527 32198 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.525990 1240 raft_consensus.cc:2726] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.526077 32262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.526273 32262 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.526376 32262 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.526427 32198 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:40.526890 1240 raft_consensus.cc:2726] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 08:14:40.528802 32262 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.529374 32262 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.529889 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.530143 32001 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.530391 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.530586 32261 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.530397 31727 catalog_manager.cc:5484] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.530690 32261 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.533126 32001 raft_consensus.cc:2396] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.534296 32261 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.535409 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.535676 32131 raft_consensus.cc:3032] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.535775 32131 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.535715 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.535924 31869 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.536074 32131 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.536482 32131 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.536512 32261 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.537539 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.537781 32261 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.537963 32261 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.538836 31869 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 2.
I20240501 08:14:40.539197 32131 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 2.
I20240501 08:14:40.539252 32199 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.539530 1240 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.539722 1240 raft_consensus.cc:686] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.539968 1240 consensus_queue.cc:237] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.540370 32001 raft_consensus.cc:1261] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 08:14:40.540475 32001 pending_rounds.cc:76] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 3
I20240501 08:14:40.540551 32001 pending_rounds.cc:97] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:40.540621 32001 raft_consensus.cc:2866] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:40.540983 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.541198 32133 raft_consensus.cc:3032] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.541510 1245 consensus_queue.cc:1035] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.541275 32133 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.542236 32133 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.547992 32133 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.550176 32261 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.551290 32133 raft_consensus.cc:2396] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.551730 31871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.551798 32198 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.551970 31871 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.551932 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.552137 32131 raft_consensus.cc:3032] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.552208 32131 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.552264 1839 raft_consensus.cc:2726] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.552453 32261 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.552488 32131 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.552769 32131 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.553387 31869 raft_consensus.cc:1261] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 08:14:40.553449 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.553647 32261 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.553750 32261 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.554036 1801 consensus_queue.cc:1035] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.554888 31871 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.555423 32199 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.555774 32131 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.556180 1839 raft_consensus.cc:2781] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.556469 1839 raft_consensus.cc:686] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.556576 32261 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.556766 1839 consensus_queue.cc:237] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.557200 32261 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.557947 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.558145 32001 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.558199 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.558372 32261 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.558478 32261 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.561015 32001 raft_consensus.cc:2396] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.562121 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.562345 32133 raft_consensus.cc:3032] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.562434 32133 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.562779 32133 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.562836 32261 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.563100 32133 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.563956 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.564143 32131 raft_consensus.cc:3032] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.564213 32131 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.564491 32131 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.564785 32131 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.565678 32261 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.567171 31731 catalog_manager.cc:5484] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 1 to 2, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 2 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.567591 32131 raft_consensus.cc:2396] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.568204 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.568456 32001 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.571784 32001 raft_consensus.cc:2396] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.572131 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.572338 32198 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.572396 32261 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.572540 32261 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.572902 1240 raft_consensus.cc:2726] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.571477 32133 raft_consensus.cc:2396] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.573012 31731 catalog_manager.cc:5484] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.573395 32198 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.573709 1240 raft_consensus.cc:2726] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.574928 32261 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.575482 32261 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 2 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.576378 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.576539 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.576623 32133 raft_consensus.cc:3032] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 08:14:40.576714 32133 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.576745 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.576787 31872 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.577068 32133 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.577440 32133 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 1 FOLLOWER]: Advancing to term 2
I20240501 08:14:40.576920 32261 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.577849 32261 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.578753 31872 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 2.
I20240501 08:14:40.579221 32199 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.579497 1240 raft_consensus.cc:2781] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 08:14:40.579640 1240 raft_consensus.cc:686] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.579981 1240 consensus_queue.cc:237] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.580307 32133 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 2.
I20240501 08:14:40.580760 32261 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.581370 32261 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.582235 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.582285 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.582435 31872 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.582515 32001 raft_consensus.cc:3032] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.582581 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:40.582619 32001 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.582773 32261 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.582878 32261 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.582890 32001 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.583288 32001 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.585278 31872 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.585619 32261 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.585793 32199 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.586092 32001 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.586287 1839 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.586792 32261 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:40.587237 1839 raft_consensus.cc:686] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.587505 1839 consensus_queue.cc:237] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.591356 31871 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.591976 1801 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.594484 31731 catalog_manager.cc:5484] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 1 to 2, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 2 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.598663 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.598897 31872 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.601331 31872 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 3.
I20240501 08:14:40.604252 31727 catalog_manager.cc:5484] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.605618 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.605886 32133 raft_consensus.cc:3032] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.605988 32133 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.606287 32133 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.606592 32133 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.606714 32001 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.607532 1245 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.612901 32199 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.613240 1839 raft_consensus.cc:2781] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.613535 1839 raft_consensus.cc:686] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.613896 1839 consensus_queue.cc:237] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.618950 31871 raft_consensus.cc:1261] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.618999 32133 raft_consensus.cc:2396] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.620937 1245 consensus_queue.cc:1035] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.621937 31731 catalog_manager.cc:5484] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 2 to 3, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 3 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.625288 32003 raft_consensus.cc:1261] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 08:14:40.625416 32003 pending_rounds.cc:76] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 3
I20240501 08:14:40.625478 32003 pending_rounds.cc:97] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:40.625545 32003 raft_consensus.cc:2866] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:40.626232 1245 consensus_queue.cc:1035] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.630033 31870 raft_consensus.cc:1261] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 08:14:40.631162 1801 consensus_queue.cc:1035] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.658464 32001 raft_consensus.cc:1261] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.659468 1245 consensus_queue.cc:1035] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.683291 31870 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.683580 31870 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.683743 31870 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.686765 31870 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.688201 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.688452 32133 raft_consensus.cc:3032] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.688580 32133 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.688759 31870 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.688897 32133 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.689276 32133 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.689684 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.690001 32001 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.691043 31870 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.691239 31870 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.691340 31870 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.692168 32133 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 3.
I20240501 08:14:40.692898 31807 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.693013 31869 raft_consensus.cc:1261] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 08:14:40.693280 1833 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.693432 1833 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.693523 1245 consensus_queue.cc:1035] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.693753 1833 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.694506 31870 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.695230 31870 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.696463 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.696676 32003 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.697014 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.697283 32261 raft_consensus.cc:3032] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.697422 32261 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.697736 32261 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.698132 32261 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.699472 32003 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.700765 31807 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.700986 31870 raft_consensus.cc:1261] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.701236 1377 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.701411 1377 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.701525 1245 consensus_queue.cc:1035] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.701802 1377 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.703904 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.704180 31868 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.704356 31868 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.704754 31867 raft_consensus.cc:1261] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 08:14:40.705475 1245 consensus_queue.cc:1035] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.708770 32000 raft_consensus.cc:1261] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.709447 1807 consensus_queue.cc:1035] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.710314 31868 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.711213 31868 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.712893 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.713214 32002 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.713419 31999 raft_consensus.cc:1261] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 08:14:40.713546 31999 pending_rounds.cc:76] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 3
I20240501 08:14:40.713629 31999 pending_rounds.cc:97] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:40.713716 31999 raft_consensus.cc:2866] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:40.714102 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.714308 32133 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.714386 32133 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.714646 1801 consensus_queue.cc:1035] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.714802 32133 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.715340 32133 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.717041 31871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.717274 31871 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.717381 31871 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.720106 32133 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.720757 32002 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.721441 32003 raft_consensus.cc:1261] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 08:14:40.721545 32003 pending_rounds.cc:76] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 3
I20240501 08:14:40.721623 32003 pending_rounds.cc:97] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:40.721693 32003 raft_consensus.cc:2866] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:40.721556 31807 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.722343 1245 consensus_queue.cc:1035] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.722410 1833 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.722623 1833 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.723009 1833 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.724758 31871 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.725703 31871 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.726634 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.726850 32262 raft_consensus.cc:3032] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.726958 32262 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.727207 32262 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.727535 32262 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.729223 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.729457 32133 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.732820 32133 raft_consensus.cc:2396] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.733222 32262 raft_consensus.cc:2396] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.734069 31807 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:40.734539 1377 raft_consensus.cc:2726] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.735275 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.735366 32261 raft_consensus.cc:2396] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:40.735486 31869 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.735584 31869 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.739176 31869 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.740278 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.740520 32133 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.743012 32001 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 3.
I20240501 08:14:40.743328 32133 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.743331 31869 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.750988 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.751374 32261 raft_consensus.cc:3032] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.751545 32261 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.752046 32261 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.752593 32261 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.757230 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.757515 31869 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.757691 31869 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.761978 31869 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.763051 31869 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.764616 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.764896 32133 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.764993 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.765267 32262 raft_consensus.cc:3032] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.765375 32262 raft_consensus.cc:729] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.765704 1761 heartbeater.cc:503] Master 127.6.56.126:36653 was elected leader, sending a full tablet report...
I20240501 08:14:40.765686 32262 consensus_queue.cc:260] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.766057 32262 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.768888 32133 raft_consensus.cc:2396] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.781173 32262 raft_consensus.cc:2396] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.782123 31807 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:40.782745 1377 raft_consensus.cc:2726] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.784482 32261 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.784993 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.785312 31807 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:40.785768 1377 raft_consensus.cc:2726] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.791796 31869 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.791966 31869 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.794921 31869 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.795625 31872 raft_consensus.cc:1226] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Rejecting Update request from peer 4827555155084237b0199283e5896782 for earlier term 3. Current term is 4. Ops: []
I20240501 08:14:40.795414 31731 catalog_manager.cc:5484] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 2 to 3, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 3 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.795970 31869 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.796217 1245 consensus_queue.cc:1046] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.796561 1245 raft_consensus.cc:3032] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.796666 1245 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.796617 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.796948 1245 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.797298 1245 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.800535 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.800796 32000 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.803601 32000 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.804343 31807 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.804489 31731 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 3 to 4, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:40.805027 1879 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.805056 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.805243 1879 raft_consensus.cc:686] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.805290 31869 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.805430 31869 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.805056 31731 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 3 to 4, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.805577 1879 consensus_queue.cc:237] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.808761 31869 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.809975 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.810082 32133 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.810238 32131 raft_consensus.cc:3032] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.810333 32131 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.810662 32131 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.810932 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.811041 32131 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.811146 32000 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.810271 31869 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.812897 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.813156 31869 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.813292 31869 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.813762 32131 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.814247 31807 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.814554 1879 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.814730 1879 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.814991 1879 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.816210 32000 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.817166 31731 catalog_manager.cc:5484] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 3 to 4, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.819351 31869 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.820744 31869 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.820719 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.820973 32131 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.821399 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.821628 32261 raft_consensus.cc:3032] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.821738 32261 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.822069 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.822007 32261 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.822290 31869 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.822352 32261 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.822459 31869 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.823935 32131 raft_consensus.cc:2396] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.825274 32261 raft_consensus.cc:2396] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.825773 31807 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:40.826243 1879 raft_consensus.cc:2726] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.826273 31869 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.826870 31869 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.827425 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.827709 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.827934 32000 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.828311 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.828559 31869 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.828691 31869 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.828930 31731 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 3 to 4, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.831050 32261 raft_consensus.cc:3032] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.831194 32261 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.831483 32261 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.831827 32261 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.830651 32000 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 3.
I20240501 08:14:40.832643 31807 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.833038 1879 raft_consensus.cc:2781] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.832232 31869 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.833453 1879 raft_consensus.cc:686] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.833817 1879 consensus_queue.cc:237] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.834839 32261 raft_consensus.cc:2396] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.835218 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.835443 32131 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.835631 31869 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.836201 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.836410 32000 raft_consensus.cc:1261] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 08:14:40.836416 32262 raft_consensus.cc:3032] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.836522 32262 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.836666 31872 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.836863 31872 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.836778 32262 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.836977 31872 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.837095 32262 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.837829 32131 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.839871 32262 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.839820 31872 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.840500 31872 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.841277 31807 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:40.841289 1839 consensus_queue.cc:1035] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.841329 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.841629 32000 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.841717 1377 raft_consensus.cc:2726] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:40.842578 31872 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.842778 31872 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.842923 31872 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.843360 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.843533 32262 raft_consensus.cc:3032] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.843621 32262 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.844478 32262 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.844770 32262 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.845374 32000 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.846189 31807 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.846531 1377 raft_consensus.cc:2781] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.846714 1377 raft_consensus.cc:686] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.847105 1377 consensus_queue.cc:237] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.849047 31872 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.849936 31872 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.850890 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.850986 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.851168 31869 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.851302 31869 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.851553 32262 raft_consensus.cc:2396] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:40.851954 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.852190 32261 raft_consensus.cc:3032] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.852325 32261 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.852653 32261 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.853029 32261 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.854847 31869 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.855578 31869 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.856717 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.857015 32131 raft_consensus.cc:3032] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.857124 32131 raft_consensus.cc:729] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.857393 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.857504 32131 consensus_queue.cc:260] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.857718 31869 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.857955 31869 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.857982 32131 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.861451 31869 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.862409 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.862725 32000 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.863996 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.864570 32262 raft_consensus.cc:3032] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.864764 32262 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.865065 32131 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 3.
I20240501 08:14:40.865445 31869 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.865098 32262 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.865876 32262 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.866494 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.866786 32003 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.870270 32003 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.865835 31807 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.872802 32262 raft_consensus.cc:2396] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:40.873626 31731 catalog_manager.cc:5484] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 2 to 3, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 3 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.876282 31807 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.879833 1377 raft_consensus.cc:2781] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.880024 1377 raft_consensus.cc:686] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.880345 1377 consensus_queue.cc:237] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.880786 32000 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 3.
I20240501 08:14:40.881307 1377 raft_consensus.cc:2781] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.881458 1377 raft_consensus.cc:686] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.880779 31731 catalog_manager.cc:5484] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 3 to 4, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.881709 1377 consensus_queue.cc:237] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:40.881927 32261 raft_consensus.cc:2396] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.882495 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.882573 31807 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:40.882709 31869 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.882855 31869 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.882896 1377 raft_consensus.cc:2726] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240501 08:14:40.885555 31869 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.887607 31869 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.888459 31728 catalog_manager.cc:5484] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 2 to 3, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 3 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.889014 31728 catalog_manager.cc:5484] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 3 to 4, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:40.889523 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.889845 32131 raft_consensus.cc:3032] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.889937 32131 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.891022 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93a196489f264058a064dc84690ba947"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.891077 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.891252 31869 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.891307 32000 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.891352 31869 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.891541 32131 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.891923 32131 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.893724 31869 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.894404 31869 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.894824 32133 raft_consensus.cc:1261] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 08:14:40.895004 32133 pending_rounds.cc:76] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 3
I20240501 08:14:40.895082 32133 pending_rounds.cc:97] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:40.895226 32133 raft_consensus.cc:2866] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:40.895954 1839 consensus_queue.cc:1035] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.896015 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.896200 32133 raft_consensus.cc:3032] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.896350 32133 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.896597 32133 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.896988 32133 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.897404 32000 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.894824 32131 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.898711 31807 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.899029 1833 raft_consensus.cc:2781] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.899163 1833 raft_consensus.cc:686] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.899199 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.899415 32003 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.899425 1833 consensus_queue.cc:237] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.901352 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.902176 31869 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.902271 31869 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.902305 31727 catalog_manager.cc:5484] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 3 to 4, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.902557 32003 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.903146 31807 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.903280 32000 raft_consensus.cc:1261] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 08:14:40.904323 32133 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.904495 1240 consensus_queue.cc:1035] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:40.906770 1377 raft_consensus.cc:2781] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.906940 1377 raft_consensus.cc:686] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.907224 1377 consensus_queue.cc:237] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.908109 31869 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.908767 31869 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.910465 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.910768 32261 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.915143 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.915388 32003 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.916581 31727 catalog_manager.cc:5484] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 3 to 4, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:40.917382 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.917609 31869 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.917707 31869 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.918350 32003 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.920730 31869 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.921334 31869 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.922108 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.922152 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.922340 32262 raft_consensus.cc:3032] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.922451 32262 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.922451 32133 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.922809 32262 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.923121 32262 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.925495 32133 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 3.
I20240501 08:14:40.925937 32262 raft_consensus.cc:2396] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.914381 32261 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.926921 31807 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.927487 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.927690 31869 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.927771 31807 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:40.927826 31869 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.929876 31869 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.930456 31869 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:40.930805 1833 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.930990 1833 raft_consensus.cc:686] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.931062 1879 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.931283 1833 consensus_queue.cc:237] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.932439 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.932632 32000 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.934499 32000 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 3.
I20240501 08:14:40.934939 31807 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.935560 1833 raft_consensus.cc:2781] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.935700 1833 raft_consensus.cc:686] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.935771 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.935958 31869 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.936066 31869 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.936182 1833 consensus_queue.cc:237] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.937775 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.938768 32133 raft_consensus.cc:3032] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.938804 31869 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.938922 32133 raft_consensus.cc:729] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:40.939442 31869 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.940147 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.940169 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.939333 32133 consensus_queue.cc:260] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.940383 32131 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.940428 32261 raft_consensus.cc:3032] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 08:14:40.940526 32261 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.942222 32261 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.942507 32261 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.943044 32131 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 3.
I20240501 08:14:40.943687 32133 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:40.944926 31807 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.945520 1377 raft_consensus.cc:2781] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:40.945806 1377 raft_consensus.cc:686] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.946049 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:40.946233 31869 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.946372 1377 consensus_queue.cc:237] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.946941 32133 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 3.
I20240501 08:14:40.947074 31869 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.950567 32261 raft_consensus.cc:2396] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:40.950841 1879 raft_consensus.cc:686] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.949751 31869 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.951174 1879 consensus_queue.cc:237] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.951984 31869 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:40.952023 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.952375 32261 raft_consensus.cc:3032] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:40.952514 32261 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:40.952667 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:40.952909 32000 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.952942 32261 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.953341 32261 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:40.955833 32000 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 4.
I20240501 08:14:40.956698 31807 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:40.957001 1902 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:40.956580 32261 raft_consensus.cc:2396] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:40.957231 1902 raft_consensus.cc:686] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.957630 1902 consensus_queue.cc:237] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:40.964094 31727 catalog_manager.cc:5484] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 2 to 4, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.964746 31727 catalog_manager.cc:5484] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 2 to 3, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 3 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.965230 31727 catalog_manager.cc:5484] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 2 to 3, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 3 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:40.965677 31727 catalog_manager.cc:5484] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 2 to 3, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 3 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:40.975164 31727 catalog_manager.cc:5484] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 3 to 4, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 4 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:40.977674 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:40.977895 32000 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.978031 32000 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:40.980587 32000 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.981266 32000 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:40.981962 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:40.982215 32133 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:40.984844 32133 raft_consensus.cc:2396] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:40.985348 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:40.985610 32261 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:40.988456 32261 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:40.988951 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:40.989207 32003 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:40.989343 32003 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:40.991832 32003 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.993100 31938 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:40.993600 1776 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:40.993799 1776 raft_consensus.cc:686] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:40.994086 1776 consensus_queue.cc:237] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:40.996544 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:40.996781 31869 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:40.996881 31869 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:40.997169 31869 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:40.997489 31869 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.000336 31869 raft_consensus.cc:2396] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.001705 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.001943 32133 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.004928 32133 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.005419 31938 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.005803 1769 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.005980 1769 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.006265 1769 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.007663 32003 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.009691 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.010298 32003 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.011121 31723 catalog_manager.cc:5484] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 5, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.012030 32003 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.015934 32003 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.018028 31723 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 4 to 5, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:41.019099 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.019325 32261 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.023269 32003 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.023891 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.024224 31869 raft_consensus.cc:3032] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.024384 31869 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.024483 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.024724 32003 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.024699 31869 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.024850 32003 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.024991 31869 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.025018 32261 raft_consensus.cc:2396] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.027027 31869 raft_consensus.cc:2396] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.026973 32003 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.027465 31939 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf; no voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.027995 1776 raft_consensus.cc:2726] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 08:14:41.028074 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.028323 32133 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.028681 32003 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.029202 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.029459 32261 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.030056 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.030236 32003 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.030352 32003 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.032021 32261 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.032414 32133 raft_consensus.cc:2396] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.032490 31938 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.033104 1776 raft_consensus.cc:2781] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.033260 1776 raft_consensus.cc:686] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.033614 1776 consensus_queue.cc:237] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.040922 32003 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.042677 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.043002 31869 raft_consensus.cc:3032] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.043112 31869 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.043567 31869 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.043949 31869 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.045394 32003 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.046779 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.047087 32000 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.047259 32000 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.048225 31869 raft_consensus.cc:2396] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.051143 32000 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.052094 32000 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.053009 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.053287 31869 raft_consensus.cc:3032] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:41.053407 31869 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.053781 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.053831 31869 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.054056 32003 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.054206 32003 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.054214 31869 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.057358 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.057659 32133 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.061133 32133 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.061844 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.062098 32133 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.062814 31869 raft_consensus.cc:2396] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.082118 32133 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 4.
I20240501 08:14:41.082896 31938 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.083441 1769 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.083645 1769 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.083822 31938 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.084064 1769 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.085471 1769 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:41.085669 1769 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.085981 1769 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.097923 31727 catalog_manager.cc:5484] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 5, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.103673 32003 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.104413 32003 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.105760 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.105973 32133 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.109077 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.109095 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.109294 32000 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.109303 32261 raft_consensus.cc:3032] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:41.109413 32261 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.109431 32000 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.109709 32261 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.110105 32261 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.112231 32000 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.113855 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.114061 32262 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.114702 32261 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 4.
I20240501 08:14:41.115172 31938 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.115505 1776 raft_consensus.cc:2781] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:41.113898 31727 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 4 to 5, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:41.115728 1776 raft_consensus.cc:686] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.116076 1776 consensus_queue.cc:237] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.116070 31727 catalog_manager.cc:5484] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 4, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 4 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.117059 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.117323 31869 raft_consensus.cc:3032] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.117420 31869 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.117720 31869 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.117990 31869 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.122254 32000 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.123332 32262 raft_consensus.cc:2396] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.124971 31869 raft_consensus.cc:2396] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.125437 31939 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf; no voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.125874 1776 raft_consensus.cc:2726] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 08:14:41.126134 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.126331 32000 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.126437 32000 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 2 FOLLOWER]: Advancing to term 3
I20240501 08:14:41.127305 31727 catalog_manager.cc:5484] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 4, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 4 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.128628 32133 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 4.
I20240501 08:14:41.131628 32000 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.132701 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.133019 31869 raft_consensus.cc:2379] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf in current term 3: Already voted for candidate d6949d828ece45dcb729633be6d49c97 in this term.
I20240501 08:14:41.133648 32000 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 3 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.134299 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.134562 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.134747 32000 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.134868 32000 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.138324 32262 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 3.
I20240501 08:14:41.138976 31938 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.139665 1914 raft_consensus.cc:2781] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Leader election won for term 3
I20240501 08:14:41.140040 1914 raft_consensus.cc:686] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 3 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.140560 1914 consensus_queue.cc:237] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.146451 31723 catalog_manager.cc:5484] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 2 to 3, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 3 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.147203 32000 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.150045 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.150328 32262 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.153870 32262 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.154335 31938 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.154461 32000 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.154688 1914 raft_consensus.cc:2781] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.154911 1914 raft_consensus.cc:686] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.155248 1914 consensus_queue.cc:237] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.155742 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.156028 32133 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.162576 32133 raft_consensus.cc:2396] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.163206 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.163462 32003 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.163628 32003 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.167217 32003 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.173059 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.173306 31869 raft_consensus.cc:3032] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.173377 31869 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.173712 31869 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.174075 31869 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.175846 31723 catalog_manager.cc:5484] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 5, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:41.178184 31869 raft_consensus.cc:2396] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.179476 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.179726 32133 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.182878 32133 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.183413 31938 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.183848 1914 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.184052 1914 raft_consensus.cc:686] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.184445 1914 consensus_queue.cc:237] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.183879 32003 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.186961 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f78273f47414fd199e593595e6aae23"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.187191 32003 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.187312 32003 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.188923 31723 catalog_manager.cc:5484] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 4 to 5, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.190896 32003 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.191579 32003 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.192793 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.193009 32262 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.193476 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.193711 32133 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.194564 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.194921 32000 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.195041 32000 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.199131 32262 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.199649 31938 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.200012 1776 raft_consensus.cc:2781] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.200177 1776 raft_consensus.cc:686] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.200501 1776 consensus_queue.cc:237] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.206071 32000 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.207273 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.207542 32262 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.212644 32133 raft_consensus.cc:2396] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.214411 32262 raft_consensus.cc:2396] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:41.215320 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.215567 31869 raft_consensus.cc:3032] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:41.215667 31869 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.215978 31869 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.216324 31869 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.219534 32113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.220038 31869 raft_consensus.cc:2396] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:41.221473 32242 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.222164 32000 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.225458 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.225744 32000 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.226230 31938 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf; no voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.226814 1914 raft_consensus.cc:2726] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240501 08:14:41.227047 32000 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.237653 32000 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.241024 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.241286 32131 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.241622 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.241958 31869 raft_consensus.cc:3032] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:41.242100 31869 raft_consensus.cc:729] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.242456 31869 consensus_queue.cc:260] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.242844 31869 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.243216 1436 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.245415 32131 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 4.
I20240501 08:14:41.245895 31869 raft_consensus.cc:2396] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.246605 31938 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.246995 1914 raft_consensus.cc:2781] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:41.247176 1914 raft_consensus.cc:686] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.247503 1914 consensus_queue.cc:237] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.249306 31723 catalog_manager.cc:5484] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 5, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.250989 1566 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.252614 32000 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.254374 1696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.254554 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e872bea48fc479686691121187aa0db"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.254768 32000 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.254940 32000 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.256942 31723 catalog_manager.cc:5484] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 4, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 4 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.257779 32000 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.262182 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.261920 32000 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.263427 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.263628 32000 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.263754 32000 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.264591 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.264807 32131 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.266611 32000 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.267534 32131 raft_consensus.cc:2396] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.267602 32262 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.267747 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.267941 32261 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.270782 32261 raft_consensus.cc:2396] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.274600 32262 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.275039 31938 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:41.275525 1776 raft_consensus.cc:2781] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.275683 1776 raft_consensus.cc:686] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.275969 1776 consensus_queue.cc:237] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.283306 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.283571 31869 raft_consensus.cc:3032] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.283663 31869 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.283969 31869 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.284327 31869 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.287330 31869 raft_consensus.cc:2396] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.288245 31939 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf; no voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.288842 32003 raft_consensus.cc:1261] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 08:14:41.289149 1769 raft_consensus.cc:2726] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 08:14:41.289641 1377 consensus_queue.cc:1035] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.290601 32000 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.292011 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.292210 32003 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.292340 32003 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.296077 32003 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.296514 32001 raft_consensus.cc:1226] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Rejecting Update request from peer d6949d828ece45dcb729633be6d49c97 for earlier term 4. Current term is 5. Ops: []
I20240501 08:14:41.297417 1879 consensus_queue.cc:1046] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.298218 32003 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.298836 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.299027 32131 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.299017 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.300819 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93a196489f264058a064dc84690ba947"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.301067 32003 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.301247 32131 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.301388 32003 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.301766 31938 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.302071 1769 raft_consensus.cc:2781] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.302244 1769 raft_consensus.cc:686] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.302553 1769 consensus_queue.cc:237] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.304695 32003 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.305527 32003 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.305776 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.305883 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.306010 31872 raft_consensus.cc:3032] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.306090 31872 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.306087 32131 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.306351 31872 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.306504 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.306634 31872 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.306679 32000 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.306767 32000 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.308881 32131 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.309254 31938 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.309561 1769 raft_consensus.cc:2781] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.309751 1769 raft_consensus.cc:686] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.309842 31872 raft_consensus.cc:2396] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.310009 1769 consensus_queue.cc:237] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.310446 32000 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.311264 32000 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.311926 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.312144 31872 raft_consensus.cc:3032] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.312156 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.312261 31872 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.312358 32000 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.312476 32000 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.312652 31872 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.312961 31872 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.314505 32000 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.315093 32000 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.315583 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.315826 32261 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.315851 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.315973 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.316144 32000 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.316255 32000 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.316191 32131 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.318657 32261 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.319142 31938 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.319428 1769 raft_consensus.cc:2781] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.319658 1769 raft_consensus.cc:686] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.319718 32131 raft_consensus.cc:2396] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.319929 32000 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.319917 1769 consensus_queue.cc:237] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.320883 32000 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.321636 31872 raft_consensus.cc:2396] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.321763 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.321982 31868 raft_consensus.cc:3032] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:41.322078 31868 raft_consensus.cc:729] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.322014 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.322265 32262 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.322338 31868 consensus_queue.cc:260] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.322647 31868 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.324354 32262 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.324946 31938 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.325256 1914 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.325404 1914 raft_consensus.cc:686] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.325654 1914 consensus_queue.cc:237] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.328085 32133 raft_consensus.cc:1261] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 08:14:41.328611 1902 consensus_queue.cc:1035] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.330627 31868 raft_consensus.cc:2396] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.339219 1304 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:41.341814 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.342270 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.343114 32000 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.343592 32131 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.343890 32000 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.349944 32000 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.352464 32131 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 4.
I20240501 08:14:41.356222 31723 catalog_manager.cc:5484] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 5, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.359557 31869 raft_consensus.cc:3032] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.359684 31869 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.359966 31869 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.360281 31869 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.363205 31869 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.367233 31852 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:14:41.363458 32132 raft_consensus.cc:1226] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Rejecting Update request from peer d6949d828ece45dcb729633be6d49c97 for earlier term 4. Current term is 5. Ops: []
I20240501 08:14:41.381727 32262 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 08:14:41.381865 32262 pending_rounds.cc:76] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4: Aborting all ops after (but not including) 3
I20240501 08:14:41.381981 32262 pending_rounds.cc:97] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:41.382108 32262 raft_consensus.cc:2866] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:41.382681 32261 raft_consensus.cc:1261] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240501 08:14:41.382778 32261 pending_rounds.cc:76] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4: Aborting all ops after (but not including) 3
I20240501 08:14:41.382843 32261 pending_rounds.cc:97] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:41.382902 32261 raft_consensus.cc:2866] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:41.383955 1833 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.383960 1902 consensus_queue.cc:1035] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:14:41.383643 31938 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.388092 1776 raft_consensus.cc:2781] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:41.388392 1776 raft_consensus.cc:686] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.388716 1776 consensus_queue.cc:237] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.391019 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.391386 32261 raft_consensus.cc:1261] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 08:14:41.391539 32000 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.392429 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.392634 32261 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.394685 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:41.394901 32000 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.395071 32000 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.395949 31727 catalog_manager.cc:5484] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 4 to 5, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.396607 31727 catalog_manager.cc:5484] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 4 to 5, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:41.396982 31983 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:41.397365 31727 catalog_manager.cc:5484] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 5, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.397857 31727 catalog_manager.cc:5484] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 4 to 5, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.398690 32000 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.402837 32000 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.403537 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.403764 32262 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.406715 32262 raft_consensus.cc:2396] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.402493 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.407524 31869 raft_consensus.cc:3032] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.407615 31869 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.407935 31869 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.408263 31869 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.409318 32133 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.409536 1769 consensus_queue.cc:1035] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.410255 32261 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 5.
I20240501 08:14:41.412173 31869 raft_consensus.cc:2396] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.412591 32133 raft_consensus.cc:2396] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.412766 31939 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf; no voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.413076 1769 raft_consensus.cc:2726] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 08:14:41.413125 32131 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240501 08:14:41.413170 31727 catalog_manager.cc:5484] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 4, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 4 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:41.413659 1377 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.417213 31938 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:41.419603 1769 raft_consensus.cc:2781] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.420172 1769 raft_consensus.cc:686] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.420517 1769 consensus_queue.cc:237] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.425594 32133 raft_consensus.cc:1261] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 08:14:41.426209 1776 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.435974 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.436228 32132 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.436357 32132 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.439306 32132 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.440732 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.440894 32132 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.441036 32262 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.442086 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.442283 32131 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.442390 32131 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.444571 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.444849 31869 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.445068 32131 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.446215 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.446463 31868 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.446815 32262 raft_consensus.cc:2396] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.448556 31727 catalog_manager.cc:5484] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 3 to 5, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 5 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.448932 31869 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 5.
I20240501 08:14:41.449062 31868 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 6.
I20240501 08:14:41.449595 32069 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.449937 32069 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.450048 32262 raft_consensus.cc:1261] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 08:14:41.450275 1245 raft_consensus.cc:2781] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.450582 1776 consensus_queue.cc:1035] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.451375 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.462512 32132 raft_consensus.cc:1261] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 08:14:41.462850 32131 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.463434 1776 consensus_queue.cc:1035] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.463642 1776 consensus_queue.cc:1225] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Peer 4827555155084237b0199283e5896782 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:41.464370 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.465082 32132 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.465178 32132 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.464943 32131 pending_rounds.cc:76] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 3
I20240501 08:14:41.465909 32131 pending_rounds.cc:97] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 08:14:41.466100 32131 raft_consensus.cc:2866] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:41.468742 32132 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.469615 31868 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240501 08:14:41.469733 31868 pending_rounds.cc:76] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 4
I20240501 08:14:41.469851 31868 pending_rounds.cc:97] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240501 08:14:41.469983 31868 raft_consensus.cc:2866] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:41.470963 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.471213 32003 raft_consensus.cc:3032] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.471300 32003 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.471606 32132 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.471585 32003 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.471884 32003 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.472256 1776 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.475581 32003 raft_consensus.cc:2396] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.476392 1943 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.479130 32133 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 08:14:41.479776 1776 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.485893 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.486652 32068 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:41.486991 31868 raft_consensus.cc:1226] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Rejecting Update request from peer 6b7faa87de354980a76c80fb864f9edf for earlier term 5. Current term is 6. Ops: []
I20240501 08:14:41.487648 1776 consensus_queue.cc:1046] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.487874 1776 consensus_queue.cc:1225] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Peer d6949d828ece45dcb729633be6d49c97 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 08:14:41.488137 32000 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.488237 32000 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.488505 32000 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.488931 32000 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.491353 32000 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 6.
I20240501 08:14:41.493243 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93a196489f264058a064dc84690ba947"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.493503 32132 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.493650 32132 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.496104 32132 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.503756 1245 raft_consensus.cc:686] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.504822 1245 consensus_queue.cc:237] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.503646 1940 raft_consensus.cc:2726] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 08:14:41.506820 32132 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
W20240501 08:14:41.507436 32069 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.56.65:38515, user_credentials={real_user=slave}} blocked reactor thread for 57495.5us
I20240501 08:14:41.508126 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.508458 31868 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.510001 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.511322 31868 raft_consensus.cc:2396] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.515908 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.516235 32000 raft_consensus.cc:3032] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.516399 32000 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.516815 32000 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.517233 32000 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.522809 32000 raft_consensus.cc:2396] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.524956 32132 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.525256 32132 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.537374 32068 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.538525 1947 raft_consensus.cc:2726] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 08:14:41.539980 32133 raft_consensus.cc:1261] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 08:14:41.541563 1777 consensus_queue.cc:1035] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.541764 32263 raft_consensus.cc:1261] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240501 08:14:41.542457 1943 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.542580 1777 consensus_queue.cc:1035] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.542910 1943 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.544842 32132 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.546535 32132 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.546749 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.546977 31868 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.547437 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.547684 32000 raft_consensus.cc:3032] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.547808 32000 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.548143 32000 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.548645 32000 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.549919 31868 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 5.
I20240501 08:14:41.552897 32000 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 5.
I20240501 08:14:41.555056 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.555445 32132 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.555671 32132 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.556102 31869 raft_consensus.cc:1261] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240501 08:14:41.556212 31869 pending_rounds.cc:76] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 4
I20240501 08:14:41.556278 31869 pending_rounds.cc:97] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 08:14:41.556387 31869 raft_consensus.cc:2866] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:41.557536 1769 consensus_queue.cc:1035] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.552456 32069 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.558691 1245 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.558861 1245 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.559110 1245 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.560079 32132 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.560727 32132 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.560850 32261 raft_consensus.cc:1261] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 08:14:41.561916 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.562109 32131 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.562206 32131 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.562788 1777 consensus_queue.cc:1035] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.565287 32131 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.565934 32131 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.567037 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.567214 32131 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.567368 32131 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.571064 32131 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.571777 32131 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.572715 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.572975 32132 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.573104 32132 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.573415 31727 catalog_manager.cc:5484] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 3 to 5, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 5 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:41.573853 31727 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 6 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:41.575457 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.575698 32000 raft_consensus.cc:3032] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.575826 32000 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.575978 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.576200 32001 raft_consensus.cc:3032] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.576160 32000 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.576339 32001 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.576531 32000 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.576524 32262 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.576638 32001 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.577013 32001 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.577015 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.577195 32260 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.579614 32000 raft_consensus.cc:2396] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.579998 32260 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 6.
I20240501 08:14:41.580386 32001 raft_consensus.cc:2396] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.581000 32131 raft_consensus.cc:1226] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Rejecting Update request from peer 6b7faa87de354980a76c80fb864f9edf for earlier term 4. Current term is 5. Ops: []
I20240501 08:14:41.575796 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.581352 32003 raft_consensus.cc:3032] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.581435 32003 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.581521 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.581698 32263 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.581709 32003 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.581965 32003 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.584712 32263 raft_consensus.cc:2396] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.585525 32132 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.586190 32132 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 4 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.586763 32003 raft_consensus.cc:2396] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.586720 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.586977 31868 raft_consensus.cc:3032] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:41.587088 31868 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.587437 31868 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.587803 31868 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.588135 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.588716 32132 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.588937 32132 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.591776 32132 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.592141 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.592389 32263 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.595360 32132 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.595789 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.595989 31869 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.596024 32068 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:41.596321 1943 raft_consensus.cc:2781] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.596520 32263 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 4.
I20240501 08:14:41.596575 1943 raft_consensus.cc:686] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.596846 1943 consensus_queue.cc:237] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.597440 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.597710 32132 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.597882 32132 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.598348 32068 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:41.598549 31869 raft_consensus.cc:2396] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.598651 1943 raft_consensus.cc:2781] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:41.598804 1943 raft_consensus.cc:686] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 4 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.599074 1943 consensus_queue.cc:237] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.599262 31868 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 4.
I20240501 08:14:41.600528 32068 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:41.601150 32068 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:41.601280 32132 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.601487 1943 raft_consensus.cc:2726] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 08:14:41.601493 1245 raft_consensus.cc:2726] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 08:14:41.602488 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.602722 32003 raft_consensus.cc:3032] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.602798 32003 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.603065 32003 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.603381 32003 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.603238 31727 catalog_manager.cc:5484] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 reported cstate change: term changed from 4 to 5, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 5 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.606421 32003 raft_consensus.cc:2396] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.606812 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.606974 32000 raft_consensus.cc:3032] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.607055 32000 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.607281 32000 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.607645 32000 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.610494 32000 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 6.
I20240501 08:14:41.612887 32263 raft_consensus.cc:1261] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 08:14:41.613741 1777 consensus_queue.cc:1035] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.614629 32068 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.615134 32068 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.615475 1943 raft_consensus.cc:2726] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 08:14:41.615648 1943 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.615648 32132 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.615813 1943 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.615852 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.616087 31868 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.616084 1943 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.616801 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.616977 32132 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.617100 32132 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.618803 31868 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 6.
I20240501 08:14:41.619186 31727 catalog_manager.cc:5484] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 3 to 4, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 4 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.619819 31727 catalog_manager.cc:5484] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 6 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.620661 32132 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.624888 32132 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.625563 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.625890 32263 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.625963 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.626195 31868 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.626825 31727 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 6 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:41.628850 32263 raft_consensus.cc:2396] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.629067 31868 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 5.
I20240501 08:14:41.629770 32069 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:41.630370 1945 raft_consensus.cc:2781] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.630540 1945 raft_consensus.cc:686] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.630584 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f78273f47414fd199e593595e6aae23"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.630820 32132 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.630932 32132 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.630874 1945 consensus_queue.cc:237] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.633714 32132 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.634402 32132 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.635223 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.635434 32000 raft_consensus.cc:3032] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.635548 32000 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.635699 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.635838 32000 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.636138 32000 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.636196 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.637171 32132 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.637343 32132 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.639041 32000 raft_consensus.cc:2396] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.639755 32068 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:41.640183 1245 raft_consensus.cc:2726] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 08:14:41.640344 32132 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.641041 32132 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.641758 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.641992 32263 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.642163 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.642416 32132 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.642555 32132 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.644840 32263 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.644975 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.645292 31869 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.645334 32132 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.646423 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.646669 32263 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.649098 31869 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 5.
I20240501 08:14:41.650435 32263 raft_consensus.cc:2396] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.650380 32069 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:41.650980 1245 raft_consensus.cc:2781] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.651100 32132 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.651151 1245 raft_consensus.cc:686] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.651487 1245 consensus_queue.cc:237] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.652132 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.652319 32132 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.652514 32132 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.653450 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.653901 31869 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.655259 32132 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.656419 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.656661 31871 raft_consensus.cc:1261] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240501 08:14:41.656705 32132 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.656776 31871 pending_rounds.cc:76] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 4
I20240501 08:14:41.656859 31871 pending_rounds.cc:97] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240501 08:14:41.656950 31871 raft_consensus.cc:2866] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:41.657752 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e872bea48fc479686691121187aa0db"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.657872 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.657976 32132 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.658090 32000 raft_consensus.cc:3032] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.658131 32132 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.658205 32000 raft_consensus.cc:729] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.658512 32000 consensus_queue.cc:260] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.658871 32000 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.660707 31868 raft_consensus.cc:1261] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 08:14:41.660934 32132 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.662228 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.662438 31869 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 5.
I20240501 08:14:41.662501 32263 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.662811 31727 catalog_manager.cc:5484] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 reported cstate change: term changed from 3 to 5, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 5 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:41.662977 32069 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:41.663554 1945 raft_consensus.cc:2781] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.663836 1945 raft_consensus.cc:686] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.664155 1945 consensus_queue.cc:237] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.665114 32263 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 6.
I20240501 08:14:41.665294 32260 raft_consensus.cc:1226] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Rejecting Update request from peer 6b7faa87de354980a76c80fb864f9edf for earlier term 5. Current term is 6. Ops: []
I20240501 08:14:41.666280 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.666503 32003 raft_consensus.cc:3032] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.666580 32003 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.666858 32003 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.667219 32003 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.668365 32068 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:41.667220 1769 consensus_queue.cc:1035] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20240501 08:14:41.668890 1245 raft_consensus.cc:2781] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.669060 1245 raft_consensus.cc:686] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.669314 1245 consensus_queue.cc:237] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.671430 32003 raft_consensus.cc:2396] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.672894 32000 raft_consensus.cc:2396] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.673130 32132 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.673367 32068 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.673669 1943 raft_consensus.cc:2726] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 08:14:41.674463 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.674651 32131 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.674768 32131 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.678990 32131 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.679594 32131 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.680444 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.680667 32263 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.680913 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.680514 31727 catalog_manager.cc:5484] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 reported cstate change: term changed from 3 to 5, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 5 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.681551 32000 raft_consensus.cc:3032] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.681739 32000 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.682116 31727 catalog_manager.cc:5484] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 3 to 5, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 5 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:41.682679 32000 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.683184 32000 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.683238 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.683455 32132 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.683518 32263 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 6.
I20240501 08:14:41.683598 32132 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.686266 32068 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:41.686836 32132 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.687631 32132 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 4 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.687741 32000 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 6.
I20240501 08:14:41.688064 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.688306 31868 raft_consensus.cc:3032] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:41.688407 31868 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:41.688709 31868 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.688839 32131 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.689091 1945 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.689101 31868 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.689231 1945 raft_consensus.cc:686] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.689476 32131 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.689682 32131 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.689481 1945 consensus_queue.cc:237] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.689659 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.690474 32261 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.691831 31868 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 4.
I20240501 08:14:41.692411 32069 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.692561 32131 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.692682 1945 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:41.692837 1945 raft_consensus.cc:686] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 4 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.693078 1945 consensus_queue.cc:237] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.694265 32131 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.694540 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.694779 31868 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.696012 32261 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 4.
I20240501 08:14:41.696373 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.696573 32000 raft_consensus.cc:3032] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.696686 32000 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.696696 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.696882 32132 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.696990 32132 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.696990 32000 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.697345 32000 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.698940 31868 raft_consensus.cc:2396] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.700134 32000 raft_consensus.cc:2396] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.701548 32132 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.702750 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.702756 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.702996 32261 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.702997 31869 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.703126 32263 raft_consensus.cc:1261] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 08:14:41.703780 1914 consensus_queue.cc:1035] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.706110 32261 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.706337 31723 catalog_manager.cc:5484] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 6 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.706616 32132 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.707584 31869 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 5.
I20240501 08:14:41.707690 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.707880 32132 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.708004 32132 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.708173 32069 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:41.708828 1245 raft_consensus.cc:2781] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.709022 1245 raft_consensus.cc:686] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.709363 1245 consensus_queue.cc:237] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.711854 32068 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.715677 31723 catalog_manager.cc:5484] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 6 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.716270 31723 catalog_manager.cc:5484] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 3 to 5, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 5 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.716882 31723 catalog_manager.cc:5484] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 reported cstate change: term changed from 3 to 4, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 4 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.717402 1245 raft_consensus.cc:2726] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20240501 08:14:41.721395 32132 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.723301 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.723567 32000 raft_consensus.cc:3032] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.723714 32000 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.724026 32000 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.724426 32000 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.727442 32000 raft_consensus.cc:2396] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.728475 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:41.728698 32261 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.730778 32132 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:41.731833 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:41.732084 32132 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.732268 32132 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.733990 32261 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 6.
I20240501 08:14:41.734544 32068 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:41.734862 1245 raft_consensus.cc:2781] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.734989 1245 raft_consensus.cc:686] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.735249 1245 consensus_queue.cc:237] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.738981 32132 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.740095 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.740360 31868 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.741705 31723 catalog_manager.cc:5484] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 6 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.743079 31868 raft_consensus.cc:2396] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.744906 32132 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.756712 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.756961 32000 raft_consensus.cc:3032] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.757072 32000 raft_consensus.cc:729] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.757334 32000 consensus_queue.cc:260] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.757637 32000 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.759667 32000 raft_consensus.cc:2396] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.760279 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.760450 32068 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.760550 32263 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.760697 32263 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.760989 1245 raft_consensus.cc:2726] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 08:14:41.763487 32263 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.764688 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.764905 31868 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.767935 31868 raft_consensus.cc:2396] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 08:14:41.772161 32263 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.773576 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.773837 32000 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.776675 32000 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.777127 32198 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.776759 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.778527 32263 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.778725 32263 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.778574 1843 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.779387 1843 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.779711 1843 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.782824 31868 raft_consensus.cc:1261] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240501 08:14:41.783142 31868 pending_rounds.cc:76] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 4
I20240501 08:14:41.783280 31868 pending_rounds.cc:97] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 08:14:41.783516 31868 raft_consensus.cc:2866] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:41.784401 1914 consensus_queue.cc:1035] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.790962 32261 raft_consensus.cc:1261] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 08:14:41.792032 1914 consensus_queue.cc:1035] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:41.793885 31723 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 4 to 6, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.794368 32263 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.795696 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.796012 31868 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.796085 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.796383 32132 raft_consensus.cc:3032] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.796519 32132 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.796818 32132 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.797147 32132 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.798930 31868 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.799180 32263 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.799404 32199 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.799773 1843 raft_consensus.cc:2781] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.800083 1843 raft_consensus.cc:686] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.800506 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.800669 32263 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.800801 32263 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.801019 32132 raft_consensus.cc:2396] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.801021 1843 consensus_queue.cc:237] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.804109 32263 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.805491 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.810160 32000 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.810611 32198 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.810956 1843 raft_consensus.cc:2781] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.811149 1843 raft_consensus.cc:686] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.811369 32263 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.811596 1843 consensus_queue.cc:237] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.811758 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.812232 32132 raft_consensus.cc:2379] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 in current term 6: Already voted for candidate 4827555155084237b0199283e5896782 in this term.
I20240501 08:14:41.812392 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.812603 32263 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.812727 32263 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.815007 32263 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.815600 32263 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.816088 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.816162 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.816383 32132 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.816565 32000 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.816601 32260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.816792 32260 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.816910 32260 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.819346 32132 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.819520 32000 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.819535 32260 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.819814 32198 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:41.820137 1843 raft_consensus.cc:2781] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.820329 1843 raft_consensus.cc:686] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.820600 1843 consensus_queue.cc:237] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.820756 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.820936 32000 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.821045 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.821266 31868 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.823537 31868 raft_consensus.cc:2396] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 08:14:41.823838 32260 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.824267 32000 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.824987 32198 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.825177 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.825384 32263 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.825450 1843 raft_consensus.cc:2781] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.825527 32263 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.825737 1843 raft_consensus.cc:686] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.826066 1843 consensus_queue.cc:237] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.829105 32263 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.830696 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.831002 31868 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.835059 31868 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 4.
I20240501 08:14:41.835789 32199 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.836261 1240 raft_consensus.cc:2781] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 08:14:41.841411 31727 catalog_manager.cc:5484] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 5 to 6, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:41.843219 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.843525 32000 raft_consensus.cc:3032] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 3 LEADER]: Stepping down as leader of term 3
I20240501 08:14:41.843638 32000 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.844035 32000 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.844552 32000 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 3 FOLLOWER]: Advancing to term 4
I20240501 08:14:41.849396 32263 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 4 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.850291 32000 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 4.
I20240501 08:14:41.850822 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.851181 32263 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.851392 32263 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.853904 1240 raft_consensus.cc:686] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 4 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.854411 1240 consensus_queue.cc:237] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.856379 32263 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.856972 32263 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.857635 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.858023 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f78273f47414fd199e593595e6aae23"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.858215 32261 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.858348 32261 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.859131 31727 catalog_manager.cc:5484] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 4 to 6, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:41.859650 31727 catalog_manager.cc:5484] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.860109 32000 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.860071 31727 catalog_manager.cc:5484] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 3 to 4, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 4 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.860565 32198 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.860498 31727 catalog_manager.cc:5484] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 4 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.860888 1843 raft_consensus.cc:2781] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.861047 1843 raft_consensus.cc:686] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.861218 32261 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.861305 1843 consensus_queue.cc:237] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:41.861775 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.862061 32132 raft_consensus.cc:2379] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 in current term 6: Already voted for candidate 4827555155084237b0199283e5896782 in this term.
I20240501 08:14:41.862280 32261 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.863066 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.863348 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.863590 32132 raft_consensus.cc:2379] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 in current term 6: Already voted for candidate 4827555155084237b0199283e5896782 in this term.
I20240501 08:14:41.863898 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.864096 32261 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.864207 32261 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.865446 32000 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.865809 32198 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:41.866102 1843 raft_consensus.cc:2781] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.866254 1843 raft_consensus.cc:686] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.866508 1843 consensus_queue.cc:237] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.867396 32261 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.867990 32261 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 5 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.867825 31727 catalog_manager.cc:5484] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:41.868458 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.868666 31868 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.870510 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.870703 32000 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.871510 31868 raft_consensus.cc:2396] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:41.872908 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.873082 32261 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.873193 32261 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.873924 31727 catalog_manager.cc:5484] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.875581 32000 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 5.
I20240501 08:14:41.875808 32261 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.876216 32198 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:41.876526 32261 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.877084 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.877302 32132 raft_consensus.cc:3032] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.877395 32132 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.877677 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.877712 32132 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.877854 32261 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.877959 32261 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.877902 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.878031 32132 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.878126 31868 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.880803 32261 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.881048 32132 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.881095 31868 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.881537 32199 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.881847 1240 raft_consensus.cc:2781] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.882016 1240 raft_consensus.cc:686] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 5 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.881980 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.882077 1843 raft_consensus.cc:2781] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.882225 32003 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.882265 1843 raft_consensus.cc:686] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.882869 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.883091 31868 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.883536 1843 consensus_queue.cc:237] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.885622 32003 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.886032 32198 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:41.886350 1978 raft_consensus.cc:2781] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.886583 1978 raft_consensus.cc:686] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.887019 1978 consensus_queue.cc:237] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.891280 31868 raft_consensus.cc:2396] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 08:14:41.882375 32261 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.882315 1240 consensus_queue.cc:237] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.895298 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.895524 32261 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.895733 32261 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.899546 32261 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.900456 32261 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.900955 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.901240 32132 raft_consensus.cc:3032] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.901329 32132 raft_consensus.cc:729] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.901603 32132 consensus_queue.cc:260] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.901734 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.901927 32132 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.901943 31868 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.902576 31727 catalog_manager.cc:5484] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 3 to 5, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 5 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.903060 31727 catalog_manager.cc:5484] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 5 to 6, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.904665 31868 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.905097 32199 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.905499 1843 raft_consensus.cc:2781] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.905741 1843 raft_consensus.cc:686] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.906034 1843 consensus_queue.cc:237] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:41.906668 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e872bea48fc479686691121187aa0db"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.906859 32132 raft_consensus.cc:2396] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.908321 32261 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.908445 32261 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.912706 32261 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.914255 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.914521 32003 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.918435 32003 raft_consensus.cc:2396] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:41.920161 32261 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.920271 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.920522 32132 raft_consensus.cc:3032] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:41.920600 32132 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.920894 32132 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.921178 32132 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.921705 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.922099 32261 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.922315 32261 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.924008 32132 raft_consensus.cc:2396] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:41.924486 32198 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:41.924878 1978 raft_consensus.cc:2726] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:41.927451 32261 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.928632 31727 catalog_manager.cc:5484] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 5 to 6, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:41.929291 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.929153 31727 catalog_manager.cc:5484] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 4 to 6, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:41.929486 31868 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.931974 31868 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.933051 32199 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.933336 1843 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.933633 1843 raft_consensus.cc:686] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.934418 32261 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.935056 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.935297 32003 raft_consensus.cc:3032] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.935369 32003 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:41.935223 1843 consensus_queue.cc:237] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.935631 32003 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:41.936012 32003 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.938238 32003 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.939188 31723 catalog_manager.cc:5484] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 5 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:41.940447 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.940665 32261 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.940773 32261 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.943496 32261 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.944144 32261 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.944629 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.944876 32132 raft_consensus.cc:3032] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:41.944972 32132 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.944765 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.945312 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.945272 32132 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.945480 32261 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.945541 32132 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.945593 32261 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.945600 32003 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.948247 32261 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.948534 32132 raft_consensus.cc:2396] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:41.948933 32261 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:41.949472 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.949580 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.949723 31868 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.949767 32132 raft_consensus.cc:3032] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:41.949851 32132 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.950163 32132 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.950381 32132 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:41.952265 32132 raft_consensus.cc:2396] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.952330 31868 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 5.
I20240501 08:14:41.952692 32003 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 7.
I20240501 08:14:41.952941 32199 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:41.953109 32198 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:41.953441 1978 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:41.953588 1978 raft_consensus.cc:686] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 5 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.953722 1240 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:41.953770 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.954010 32261 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.953940 1978 consensus_queue.cc:237] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.954098 32261 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.953944 1240 raft_consensus.cc:686] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 7 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.955186 1240 consensus_queue.cc:237] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.956701 32261 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.957664 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:41.957890 31868 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.958065 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.958289 32132 raft_consensus.cc:3032] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:41.958377 32132 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.958667 32132 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.958972 32132 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:41.959175 31723 catalog_manager.cc:5484] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 4 to 5, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 5 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.959702 31723 catalog_manager.cc:5484] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 6 to 7, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 7 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.957231 32261 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:41.960677 31868 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:41.961643 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.961831 32261 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.961901 32199 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:41.961966 32261 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.961982 32132 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:41.962813 32003 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 08:14:41.963541 1843 raft_consensus.cc:2781] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:41.963500 1945 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:41.963773 1843 raft_consensus.cc:686] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:41.964046 1843 consensus_queue.cc:237] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.964861 32000 raft_consensus.cc:1261] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:41.965986 1245 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:41.969225 31868 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 08:14:41.969833 1945 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:41.972337 31869 raft_consensus.cc:1261] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:41.973726 1245 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:41.973955 1245 consensus_queue.cc:1225] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Peer d6949d828ece45dcb729633be6d49c97 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 08:14:41.975479 31868 pending_rounds.cc:76] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 4
I20240501 08:14:41.975570 31868 pending_rounds.cc:97] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 08:14:41.975664 31868 raft_consensus.cc:2866] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:41.975776 32261 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.976979 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.977236 32132 raft_consensus.cc:3032] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:41.977350 32132 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.977644 32132 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.977941 32132 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.984390 32132 raft_consensus.cc:2396] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:41.986382 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.986523 32261 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.986699 32000 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.987896 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:41.988159 32261 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:41.988308 32261 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.990653 32000 raft_consensus.cc:2396] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:41.991437 32198 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:41.991884 1240 raft_consensus.cc:2726] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:41.993387 32261 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.994103 32261 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:41.994547 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:41.994642 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:41.994797 32132 raft_consensus.cc:3032] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:41.994901 32132 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:41.995049 32000 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.995219 32132 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:41.995527 32132 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:41.996729 31723 catalog_manager.cc:5484] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 5 to 6, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:41.998317 32000 raft_consensus.cc:2396] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:41.999733 32132 raft_consensus.cc:2396] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.000228 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:42.000237 32198 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:42.000465 32261 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.000607 32261 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:42.000649 1843 raft_consensus.cc:2726] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:42.002699 32261 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.003238 32261 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.003859 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.003763 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.004130 32132 raft_consensus.cc:3032] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:42.004211 31868 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:42.004215 32132 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.004498 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:42.004746 32261 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.004529 32132 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.004951 32132 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:42.004936 32261 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.007521 31868 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 5.
I20240501 08:14:42.008011 32132 raft_consensus.cc:2396] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:42.008023 32199 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.008548 1843 raft_consensus.cc:2781] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:42.008729 1843 raft_consensus.cc:686] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 5 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.008991 1843 consensus_queue.cc:237] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.009411 32261 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.010553 32261 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.010607 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.010809 31868 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.011847 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:42.012059 32261 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.012197 32261 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.012733 31723 catalog_manager.cc:5484] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 4 to 5, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 5 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.014401 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.014751 32000 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.014920 32261 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.015767 31868 raft_consensus.cc:2396] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 08:14:42.016213 32261 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.016683 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.016906 31869 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.018821 32000 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:42.019023 31869 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 6.
I20240501 08:14:42.019184 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.019407 32132 raft_consensus.cc:3032] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:42.019430 32199 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.019498 32132 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.019943 32198 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:42.020237 1240 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:42.020398 1240 raft_consensus.cc:686] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.020725 1843 raft_consensus.cc:2781] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:42.020642 1240 consensus_queue.cc:237] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.020892 1843 raft_consensus.cc:686] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.021181 1843 consensus_queue.cc:237] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.019805 32132 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.022930 32132 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.026858 32132 raft_consensus.cc:2396] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:42.027912 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.028167 31869 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.028340 31869 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.031013 31869 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.032280 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.032536 32132 raft_consensus.cc:3032] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:42.032618 32132 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.032902 32132 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.033246 32132 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.034016 31727 catalog_manager.cc:5484] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 4 to 6, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.034662 31727 catalog_manager.cc:5484] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 5 to 6, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 6 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:42.035823 32132 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 6.
I20240501 08:14:42.036868 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.037063 32000 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.037320 31807 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.037624 1377 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:42.037798 1377 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.037633 31869 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.038064 1377 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.039572 32000 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 6.
I20240501 08:14:42.040848 31871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.041047 31871 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.041172 31871 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.042012 31868 raft_consensus.cc:1261] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.042306 31723 catalog_manager.cc:5484] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 5 to 6, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 6 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.043752 1245 consensus_queue.cc:1035] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.043751 31871 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.044831 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.045056 32261 raft_consensus.cc:3032] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.045151 32261 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.045467 32261 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.045799 32261 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.045909 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.046119 32000 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.046903 31871 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.047771 32003 raft_consensus.cc:1261] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.048346 1943 consensus_queue.cc:1035] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.048684 32261 raft_consensus.cc:2396] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 08:14:42.050235 32000 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.050608 31807 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:42.051179 1996 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.051342 1996 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.051630 1996 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.054047 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.054286 31868 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.054455 31868 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.058133 31868 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.058980 31868 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.060083 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.060343 32000 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.060443 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.060628 32132 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.060693 32132 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.060966 32132 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.061336 32132 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.063135 32000 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.063355 32132 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.064394 31807 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.064864 1996 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.065011 1996 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.065229 1996 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.065321 31727 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 6 to 7, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 7 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.066774 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.066962 31868 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.067046 31868 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.069850 31868 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.071991 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.072214 32261 raft_consensus.cc:3032] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.072322 32261 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.074265 32261 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.074584 32261 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.075078 31728 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 6 to 7, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 7 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:42.085038 32261 raft_consensus.cc:2396] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.090039 31868 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.093389 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.093665 32132 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.095407 31871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.095649 31871 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.095754 31871 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.097307 32132 raft_consensus.cc:2396] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.097903 31807 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:42.098379 1377 raft_consensus.cc:2726] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:42.098577 31871 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.099397 31871 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.099903 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.100100 32132 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.100798 31871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.101017 31871 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.101162 31871 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.102996 32132 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.103941 31871 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.105106 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.105336 32263 raft_consensus.cc:3032] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.105433 32263 raft_consensus.cc:729] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.105700 32263 consensus_queue.cc:260] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.106003 32263 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.109381 32263 raft_consensus.cc:2396] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.110325 31871 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.110716 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.110945 32132 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.111362 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.111563 31869 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.111665 31869 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.113819 32132 raft_consensus.cc:2396] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.114378 31869 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.114989 31869 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.115788 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.115885 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.115983 32132 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.116086 32000 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.118914 32000 raft_consensus.cc:2396] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 08:14:42.119673 31807 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:42.120124 32132 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.120496 1377 raft_consensus.cc:2726] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:42.121136 31807 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:42.121461 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.121779 31869 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.121938 31869 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.124924 31869 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.125232 1377 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.125540 1377 raft_consensus.cc:686] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.125798 1377 consensus_queue.cc:237] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.129029 31723 catalog_manager.cc:5484] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 5 to 7, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 7 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.131237 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.131494 32000 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.131886 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.132146 32132 raft_consensus.cc:3032] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.132232 32132 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.130594 31869 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.133221 32132 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.133611 32132 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.134222 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.134519 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.134755 31869 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.134971 32261 raft_consensus.cc:3032] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.135124 31869 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.135241 32261 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.138065 31869 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.140058 32132 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.140576 32261 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.141237 32261 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.144347 32261 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.140146 32000 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.141005 31807 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.145145 1377 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.145293 1377 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.145540 1377 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.149587 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.149880 32132 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.149605 31727 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 6 to 7, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 7 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:42.150028 31869 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.151065 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.151289 31868 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.151466 31868 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.153473 32132 raft_consensus.cc:2396] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.149711 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.155483 32263 raft_consensus.cc:3032] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.155648 32263 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.156005 32263 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.156456 32263 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.159894 32263 raft_consensus.cc:2396] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.160444 31807 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:42.160887 1996 raft_consensus.cc:2726] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:42.161826 31807 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:42.162205 1996 raft_consensus.cc:2726] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:42.162788 31868 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.163928 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.164175 32261 raft_consensus.cc:3032] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:42.164350 32261 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.164676 32261 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.168501 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.168949 32000 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.171739 32261 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.171931 31868 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.174001 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.174268 31869 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.174460 31869 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.177251 31869 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.177954 32000 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 6.
I20240501 08:14:42.178920 31807 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.180922 1996 raft_consensus.cc:2781] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:42.181053 1996 raft_consensus.cc:686] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.181322 1996 consensus_queue.cc:237] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.190279 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.190534 32132 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.192245 32261 raft_consensus.cc:2396] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:42.193557 32132 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.193598 31869 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.195492 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.195739 31868 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.195888 31868 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.193253 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.202973 32263 raft_consensus.cc:3032] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.203106 32263 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.203595 32263 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.204135 32263 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.209602 32263 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:42.212882 31807 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:42.213294 1377 raft_consensus.cc:2726] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:42.221861 32132 raft_consensus.cc:1261] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.222563 1843 consensus_queue.cc:1035] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:42.222810 1843 consensus_queue.cc:1225] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer 4827555155084237b0199283e5896782 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:42.224606 32132 pending_rounds.cc:76] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 3
I20240501 08:14:42.224701 32132 pending_rounds.cc:97] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 08:14:42.224779 32132 raft_consensus.cc:2866] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.226902 1436 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.228628 32000 raft_consensus.cc:1261] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.229168 1843 consensus_queue.cc:1035] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.234788 32113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.235282 31983 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.202097 31868 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.242797 31868 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.244647 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.245384 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.245658 32261 raft_consensus.cc:3032] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.245759 32261 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.246383 32242 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240501 08:14:42.248271 31727 catalog_manager.cc:5484] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 5 to 6, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 6 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.256651 32261 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.257015 32261 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.258494 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.258782 31868 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.258919 31868 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:42.265877 32261 raft_consensus.cc:2396] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 08:14:42.268893 32000 raft_consensus.cc:1261] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.269410 1978 consensus_queue.cc:1035] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:42.270893 31868 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.273275 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.273538 32000 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:42.276547 32000 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 5.
I20240501 08:14:42.276957 31807 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.277297 1996 raft_consensus.cc:2781] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 08:14:42.277817 1996 raft_consensus.cc:686] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 5 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.278085 1996 consensus_queue.cc:237] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.281365 31731 catalog_manager.cc:5484] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 4 to 5, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 5 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.285097 31807 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:42.285717 1996 raft_consensus.cc:2726] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:42.287699 32001 raft_consensus.cc:1261] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.288396 1978 consensus_queue.cc:1035] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:42.291868 31869 raft_consensus.cc:1226] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 5 LEADER]: Rejecting Update request from peer 64f7786244bd4f90af1ad476966482c4 for earlier term 4. Current term is 5. Ops: []
I20240501 08:14:42.292816 1978 consensus_queue.cc:1046] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.293052 1978 consensus_queue.cc:1225] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer d6949d828ece45dcb729633be6d49c97 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20240501 08:14:42.293164 1978 raft_consensus.cc:3032] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 08:14:42.293254 1978 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.293496 1978 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.293851 1978 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:42.300822 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.301595 32261 raft_consensus.cc:2396] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 08:14:42.302565 31868 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 5 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.305260 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.305519 31869 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.305708 31869 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 4 FOLLOWER]: Advancing to term 5
I20240501 08:14:42.308799 31869 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.310042 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.310441 32132 raft_consensus.cc:2379] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 in current term 5: Already voted for candidate 4827555155084237b0199283e5896782 in this term.
I20240501 08:14:42.311064 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.311358 32002 raft_consensus.cc:2396] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:42.312288 31807 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:42.316433 1996 raft_consensus.cc:2726] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20240501 08:14:42.321174 31869 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 5 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.324159 1304 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.325850 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.326148 31869 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.326280 31869 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.341735 32132 raft_consensus.cc:1261] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.342782 32002 raft_consensus.cc:1261] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.343506 1979 consensus_queue.cc:1035] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:42.343734 1979 consensus_queue.cc:1225] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer 4827555155084237b0199283e5896782 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:42.344839 1979 consensus_queue.cc:1035] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.345609 32132 pending_rounds.cc:76] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 3
I20240501 08:14:42.345719 32132 pending_rounds.cc:97] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 08:14:42.345803 32132 raft_consensus.cc:2866] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.350229 31999 raft_consensus.cc:1261] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.351956 1979 consensus_queue.cc:1035] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:42.365960 31998 raft_consensus.cc:1261] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 08:14:42.367504 1240 consensus_queue.cc:1035] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.369352 32132 raft_consensus.cc:1261] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.370128 1978 consensus_queue.cc:1035] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:42.370362 1978 consensus_queue.cc:1225] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer 4827555155084237b0199283e5896782 log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:42.371591 32132 pending_rounds.cc:76] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 3
I20240501 08:14:42.371789 32132 pending_rounds.cc:97] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 08:14:42.371961 32132 raft_consensus.cc:2866] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.374380 32131 raft_consensus.cc:1261] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20240501 08:14:42.374496 32131 pending_rounds.cc:76] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 5
I20240501 08:14:42.374570 32131 pending_rounds.cc:97] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20240501 08:14:42.374639 32131 raft_consensus.cc:2866] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.377091 1843 consensus_queue.cc:1035] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.395911 31869 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.402334 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.402479 31868 raft_consensus.cc:1226] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Rejecting Update request from peer 64f7786244bd4f90af1ad476966482c4 for earlier term 6. Current term is 7. Ops: []
I20240501 08:14:42.402616 32261 raft_consensus.cc:3032] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.402711 32261 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.403023 32261 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.403373 32261 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.403308 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.405786 31852 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 08:14:42.404356 32000 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.406411 1696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.406584 32261 raft_consensus.cc:2396] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 08:14:42.402302 31869 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.414975 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.415243 31868 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.415392 31868 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.418944 32132 raft_consensus.cc:1261] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.421171 1978 consensus_queue.cc:1035] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.421386 31868 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.422297 31868 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.422683 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.422911 32002 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.424387 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.424629 32132 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.427016 32002 raft_consensus.cc:2396] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 08:14:42.427991 31871 raft_consensus.cc:1261] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.428123 32000 raft_consensus.cc:2396] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 08:14:42.428663 31807 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:42.428695 1240 consensus_queue.cc:1035] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.429111 1996 raft_consensus.cc:2726] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:42.436648 1566 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:42.441869 32000 raft_consensus.cc:1261] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.443027 1978 consensus_queue.cc:1035] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.443528 32132 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.444036 31807 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:42.444422 1377 raft_consensus.cc:2781] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.444587 1377 raft_consensus.cc:686] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.444866 1377 consensus_queue.cc:237] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.448820 32002 raft_consensus.cc:1261] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.449374 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93a196489f264058a064dc84690ba947"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.449635 31868 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.449882 31868 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.451205 31728 catalog_manager.cc:5484] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 5 to 7, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 7 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.452764 31868 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.453164 1978 consensus_queue.cc:1035] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.460338 31868 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.460829 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.461122 32132 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.461387 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.461582 31868 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.461848 31868 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.463137 31871 raft_consensus.cc:1261] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.463933 32132 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.463896 1978 consensus_queue.cc:1035] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.464082 1978 consensus_queue.cc:1225] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer d6949d828ece45dcb729633be6d49c97 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 08:14:42.464340 31807 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.464641 1377 raft_consensus.cc:2781] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.464781 1377 raft_consensus.cc:686] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.465076 1377 consensus_queue.cc:237] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.465345 31871 pending_rounds.cc:76] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 4
I20240501 08:14:42.465451 31871 pending_rounds.cc:97] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 08:14:42.465528 31871 raft_consensus.cc:2866] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.468663 31869 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20240501 08:14:42.469234 1978 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.472278 31868 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.473044 32131 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240501 08:14:42.474041 1996 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:42.475394 31868 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.478242 31872 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.484819 31727 catalog_manager.cc:5484] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 5 to 7, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 7 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:42.492198 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.492475 32261 raft_consensus.cc:3032] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.492558 32261 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.492839 32261 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.493220 32261 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.497455 32261 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.502760 31807 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.503158 1377 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.504019 32131 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240501 08:14:42.504180 32131 pending_rounds.cc:76] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 4
I20240501 08:14:42.504247 32131 pending_rounds.cc:97] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 08:14:42.504349 32131 raft_consensus.cc:2866] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.506053 1240 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.506955 1377 raft_consensus.cc:686] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.507324 1377 consensus_queue.cc:237] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.459985 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.487076 32001 raft_consensus.cc:1261] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 08:14:42.517145 32000 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.473923 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.485173 32003 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20240501 08:14:42.517606 32002 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.517731 1377 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:42.518189 1377 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:42.521973 32002 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.524175 32131 raft_consensus.cc:1261] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 08:14:42.525302 1377 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:42.525578 32261 raft_consensus.cc:1261] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20240501 08:14:42.526407 32261 pending_rounds.cc:76] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4: Aborting all ops after (but not including) 5
I20240501 08:14:42.526510 32261 pending_rounds.cc:97] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20240501 08:14:42.526612 32261 raft_consensus.cc:2866] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.527915 1377 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.528177 1377 consensus_queue.cc:1225] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Peer 64f7786244bd4f90af1ad476966482c4 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 08:14:42.530673 32000 raft_consensus.cc:2396] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 08:14:42.531562 31872 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.531736 31872 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.532528 32000 raft_consensus.cc:1261] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.534610 31872 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.534672 1978 consensus_queue.cc:1035] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 08:14:42.536266 31872 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.537855 31872 raft_consensus.cc:1226] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Rejecting Update request from peer 64f7786244bd4f90af1ad476966482c4 for earlier term 5. Current term is 6. Ops: []
I20240501 08:14:42.538390 1843 consensus_queue.cc:1046] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.538666 1843 raft_consensus.cc:3032] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:42.538764 1843 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.539016 1843 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.539388 1843 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.539461 32132 raft_consensus.cc:1261] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 08:14:42.538589 32000 raft_consensus.cc:1261] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 08:14:42.539999 1377 consensus_queue.cc:1035] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.540166 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.540393 31869 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.540504 31869 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.541002 1377 consensus_queue.cc:1035] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.541159 1377 consensus_queue.cc:1225] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Peer 6b7faa87de354980a76c80fb864f9edf log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 08:14:42.541847 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.542068 32133 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.542286 32261 pending_rounds.cc:76] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4: Aborting all ops after (but not including) 4
I20240501 08:14:42.542371 32261 pending_rounds.cc:97] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240501 08:14:42.542424 32261 raft_consensus.cc:2866] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.542666 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.543352 32000 pending_rounds.cc:76] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 4
I20240501 08:14:42.543442 32000 pending_rounds.cc:97] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240501 08:14:42.543511 32000 raft_consensus.cc:2866] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.546262 31869 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.548480 31869 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.548686 32003 raft_consensus.cc:1261] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 08:14:42.550658 32133 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 6.
I20240501 08:14:42.551441 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.551728 32003 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.553243 32263 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 6.
I20240501 08:14:42.555807 32003 raft_consensus.cc:2396] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240501 08:14:42.557170 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.557399 32133 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.557547 2047 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:42.560146 32133 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 6.
I20240501 08:14:42.562347 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.562618 31869 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.562788 31869 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.566582 31869 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.569363 31869 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.569718 31807 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.570034 31807 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:42.570045 2048 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:42.570374 2048 raft_consensus.cc:686] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.570449 1377 raft_consensus.cc:2781] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:42.570591 1377 raft_consensus.cc:686] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.570721 2048 consensus_queue.cc:237] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.570863 1377 consensus_queue.cc:237] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.574038 32132 raft_consensus.cc:1261] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20240501 08:14:42.575619 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.575819 32263 raft_consensus.cc:3032] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:42.575912 32263 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.576171 32263 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.576471 32263 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.577852 31869 raft_consensus.cc:1226] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Rejecting Update request from peer 64f7786244bd4f90af1ad476966482c4 for earlier term 5. Current term is 6. Ops: []
I20240501 08:14:42.579576 32263 raft_consensus.cc:2396] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:42.579960 32132 pending_rounds.cc:76] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 4
I20240501 08:14:42.580060 32132 pending_rounds.cc:97] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 08:14:42.580165 32132 raft_consensus.cc:2866] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.580317 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.580534 32133 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.581881 31728 catalog_manager.cc:5484] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 6 to 7, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 7 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.584213 32133 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 6.
I20240501 08:14:42.587239 31807 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:42.587647 1377 raft_consensus.cc:2781] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:42.587874 1377 raft_consensus.cc:686] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.587805 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:42.588033 31869 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.588163 31869 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.588119 1377 consensus_queue.cc:237] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.591401 31869 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.592237 31869 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.593247 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.593492 32263 raft_consensus.cc:3032] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.593605 32263 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.593595 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.593930 32003 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.593899 32263 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.594352 32263 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.596413 32003 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.597280 31807 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.597604 1377 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.597746 1377 raft_consensus.cc:686] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.598017 1377 consensus_queue.cc:237] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.597465 32263 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 7.
I20240501 08:14:42.598692 31731 catalog_manager.cc:5484] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 5 to 6, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 6 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.600039 31731 catalog_manager.cc:5484] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 5 to 6, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 6 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.600909 31731 catalog_manager.cc:5484] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 4 to 6, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 6 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:42.609051 31731 catalog_manager.cc:5484] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 6 to 7, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 7 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.619480 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.619727 32003 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.619856 32003 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.623093 32003 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.623819 32003 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.625247 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.625510 32132 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.625509 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.625698 32263 raft_consensus.cc:3032] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.625797 32263 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.626120 32263 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.626502 32263 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.629073 32132 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.629324 32263 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.629631 31938 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.629953 1777 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.630105 1777 raft_consensus.cc:686] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.630416 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.630681 32003 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.630784 32003 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.630385 1777 consensus_queue.cc:237] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.633672 32003 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.634742 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.634964 31869 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.635057 31869 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.634683 31731 catalog_manager.cc:5484] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 7 to 8, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.635326 31869 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.635712 31869 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.638461 31869 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.638882 31939 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.639210 1777 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.639343 1777 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.639426 32003 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.639617 1777 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.641113 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.641341 32132 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.643505 31731 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 7 to 8, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:42.644331 32132 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.645097 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.645287 32003 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.645401 32003 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.648043 32003 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.649189 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.649227 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.649410 31869 raft_consensus.cc:3032] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.649420 32261 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.649497 31869 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.649778 31869 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.650077 31869 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.652153 31869 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.652179 32261 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.652617 31939 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.652959 1777 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.653110 1777 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.653362 1777 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.654335 32003 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.655339 32263 raft_consensus.cc:1261] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20240501 08:14:42.655686 32263 pending_rounds.cc:76] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4: Aborting all ops after (but not including) 4
I20240501 08:14:42.655834 32263 pending_rounds.cc:97] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240501 08:14:42.656008 32263 raft_consensus.cc:2866] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.656126 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.656325 32003 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.656435 32003 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.656672 1377 consensus_queue.cc:1035] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:42.656869 1377 consensus_queue.cc:1225] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [LEADER]: Peer 64f7786244bd4f90af1ad476966482c4 log is divergent from this leader: its last log entry 2.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:42.657752 32263 pending_rounds.cc:76] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4: Aborting all ops after (but not including) 3
I20240501 08:14:42.657876 32263 pending_rounds.cc:97] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:42.657999 32263 raft_consensus.cc:2866] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.659091 32003 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.659688 32003 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.661645 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.661829 32000 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.661969 32000 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.662144 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.662192 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.662339 32261 raft_consensus.cc:3032] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.662410 32261 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.662437 32132 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.662669 32261 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.663008 32261 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.665529 32132 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.665712 32261 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.665927 31938 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.666249 1777 raft_consensus.cc:2781] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.666375 1777 raft_consensus.cc:686] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.666673 1777 consensus_queue.cc:237] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.666911 32000 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.667068 32002 raft_consensus.cc:1261] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20240501 08:14:42.667603 32000 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.668332 1377 consensus_queue.cc:1035] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.668412 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.668627 31869 raft_consensus.cc:3032] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.668707 31869 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.668967 31869 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.669281 31869 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.668097 32003 raft_consensus.cc:1261] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20240501 08:14:42.671974 32263 raft_consensus.cc:1261] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20240501 08:14:42.672109 32263 pending_rounds.cc:76] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4: Aborting all ops after (but not including) 5
I20240501 08:14:42.672166 32263 pending_rounds.cc:97] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20240501 08:14:42.672246 32263 raft_consensus.cc:2866] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:42.672528 31869 raft_consensus.cc:2396] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:42.672454 2047 consensus_queue.cc:1035] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:42.672605 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.672791 32003 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.672899 32003 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.673481 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.673731 32133 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.676035 32003 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.676913 2047 consensus_queue.cc:1035] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.677491 32003 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.678704 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.678956 31872 raft_consensus.cc:3032] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.679066 31872 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.679385 31872 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.679888 31872 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.680804 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.681037 32003 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.681195 32003 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.683528 31872 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.684116 32131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.684372 32131 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.684536 32003 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.685639 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.685822 32130 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.686033 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.686252 32263 raft_consensus.cc:3032] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.686331 32263 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.686625 32263 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.686941 32263 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.689093 32131 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.690312 32263 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.690425 31938 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.690847 1777 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.690990 31938 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.691079 1777 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.691231 1914 raft_consensus.cc:2781] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.691327 32003 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.691427 1777 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.692397 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.692607 32003 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.692732 32003 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.693970 32130 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.695906 32133 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.696472 31938 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:42.697058 1777 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.697149 32003 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.697190 1777 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.697885 1777 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.698518 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.700623 31731 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 7 to 8, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.702692 32263 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.703285 31938 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.703645 1777 raft_consensus.cc:2781] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.703788 1777 raft_consensus.cc:686] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.704126 1777 consensus_queue.cc:237] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.705701 32003 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.707001 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.707280 31872 raft_consensus.cc:2379] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf in current term 7: Already voted for candidate d6949d828ece45dcb729633be6d49c97 in this term.
I20240501 08:14:42.707433 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.707727 32003 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.707937 32003 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.709259 1914 raft_consensus.cc:686] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.709635 1914 consensus_queue.cc:237] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.715718 32003 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.716827 32003 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 6 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.718420 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.718653 32263 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.719522 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.719810 32002 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.719960 32002 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.721621 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.721837 31872 raft_consensus.cc:3032] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 5 LEADER]: Stepping down as leader of term 5
I20240501 08:14:42.721954 31872 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.722251 31872 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.722637 31872 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.723861 32002 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.724597 32002 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.725643 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.725669 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.725867 32132 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.725883 32261 raft_consensus.cc:3032] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.725955 32261 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.726225 32261 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.726565 32261 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.729647 32261 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.730077 32263 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 6.
I20240501 08:14:42.730115 31938 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.730422 2067 raft_consensus.cc:2781] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.730592 2067 raft_consensus.cc:686] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.730643 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.730829 32003 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.730945 32003 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.731058 32132 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.730948 2067 consensus_queue.cc:237] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.730602 31938 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.732810 2067 raft_consensus.cc:2781] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:42.732903 31872 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 6.
I20240501 08:14:42.732968 2067 raft_consensus.cc:686] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 6 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.733197 2067 consensus_queue.cc:237] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.735222 31731 catalog_manager.cc:5484] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 7, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 7 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.735987 31731 catalog_manager.cc:5484] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 7, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 7 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.736331 32003 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.736732 31731 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 7 to 8, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:42.737277 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.737466 31872 raft_consensus.cc:3032] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.737560 31872 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.737812 31872 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.737787 31731 catalog_manager.cc:5484] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 7, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 7 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.738175 31872 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.741039 31872 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.741436 31939 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.741712 1914 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.741817 1914 raft_consensus.cc:686] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.742048 1914 consensus_queue.cc:237] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.743821 32003 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.745267 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f78273f47414fd199e593595e6aae23"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.745574 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.745810 32132 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.745992 31731 catalog_manager.cc:5484] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 5 to 6, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 6 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.746505 31731 catalog_manager.cc:5484] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 7, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 7 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:42.746937 31731 catalog_manager.cc:5484] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 7 to 8, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.747370 31731 catalog_manager.cc:5484] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 7, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 7 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:42.748878 32132 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.752497 32002 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.752668 32002 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.755518 32002 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.756317 32002 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.757215 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.757354 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.757534 32261 raft_consensus.cc:3032] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.757539 32003 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.757652 32003 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.757668 32261 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:42.758102 32261 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.758510 32261 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.760413 32003 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.760854 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.761163 32132 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.761603 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.761782 32263 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.762688 32261 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.765203 32132 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.765265 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.765499 31872 raft_consensus.cc:3032] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.765604 31872 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.765904 31872 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.766186 31872 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.768127 31872 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.768532 31938 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.768568 31939 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.768884 1914 raft_consensus.cc:2781] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.769069 2067 raft_consensus.cc:2781] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.769102 1914 raft_consensus.cc:686] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.769209 2067 raft_consensus.cc:686] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.769395 1914 consensus_queue.cc:237] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.769456 2067 consensus_queue.cc:237] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.764415 32263 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.770998 32003 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.772468 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.772656 32003 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.772749 32003 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.772941 31731 catalog_manager.cc:5484] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 7, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 7 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.773480 31731 catalog_manager.cc:5484] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 7, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 7 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.775197 32003 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.776314 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.776568 31872 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.777133 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.777377 32132 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 5 FOLLOWER]: Advancing to term 6
I20240501 08:14:42.779558 31872 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 6.
I20240501 08:14:42.779817 32003 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 6 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.779953 31939 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.780478 2067 raft_consensus.cc:2781] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Leader election won for term 6
I20240501 08:14:42.780623 2067 raft_consensus.cc:686] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 6 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.780848 2067 consensus_queue.cc:237] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.782574 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e872bea48fc479686691121187aa0db"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.782783 32002 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.782967 32002 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.784776 32132 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 6.
I20240501 08:14:42.785410 31731 catalog_manager.cc:5484] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.785841 32002 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.786409 32002 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.787053 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.787317 32263 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.787683 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.787911 32002 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.788070 32002 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.790138 32263 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.790624 31938 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.791045 2067 raft_consensus.cc:2781] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.791172 2067 raft_consensus.cc:686] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.791453 2067 consensus_queue.cc:237] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.791533 32002 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.792135 32002 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.793534 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.793810 32261 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.795130 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.795379 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.795800 32132 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.795588 32002 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.795982 32002 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.795953 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.796185 31872 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.797144 32261 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.797824 31938 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.798208 1914 raft_consensus.cc:2781] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.798346 1914 raft_consensus.cc:686] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.798619 1914 consensus_queue.cc:237] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.800264 31872 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.802296 32132 raft_consensus.cc:2396] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:42.804172 32002 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.804908 32002 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.805879 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.805979 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.806149 32132 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.806113 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93a196489f264058a064dc84690ba947"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.806279 32002 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.806322 31872 raft_consensus.cc:3032] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.806397 32002 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.806506 31872 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.806923 31872 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.807400 31872 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.809202 32132 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.809146 32002 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.809643 31938 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.810007 1777 raft_consensus.cc:2781] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.810178 1777 raft_consensus.cc:686] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.810511 1777 consensus_queue.cc:237] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.811774 31872 raft_consensus.cc:2396] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 08:14:42.812067 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.812294 31869 raft_consensus.cc:3032] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.812397 31869 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.812688 31869 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.813005 31869 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.812991 31731 catalog_manager.cc:5484] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 8, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.815598 31869 raft_consensus.cc:2396] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:42.816637 32002 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.817848 32003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.817875 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.818050 32003 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.818112 32132 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.818192 32003 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.820991 32132 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.821309 32003 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.822501 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.822723 31869 raft_consensus.cc:3032] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.822805 31869 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.822710 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.822976 32261 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.823176 31869 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.823493 31869 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.823522 31731 catalog_manager.cc:5484] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 8, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:42.824047 31731 catalog_manager.cc:5484] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 7 to 8, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.825268 32003 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.826221 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.826434 32000 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.826536 32000 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.826660 32261 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.827622 31938 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.828059 31938 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:42.828550 1914 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.828733 1914 raft_consensus.cc:686] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.828752 1777 raft_consensus.cc:2781] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.829003 1914 consensus_queue.cc:237] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.829279 32000 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.829903 32000 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.830847 32001 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.831013 32001 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.831151 32001 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.833086 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.833360 32261 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.833470 32001 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.834086 32001 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:42.835196 31869 raft_consensus.cc:2396] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:42.835665 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.835775 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.835975 32132 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.836108 32133 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.837023 32261 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.837607 31938 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.838059 2067 raft_consensus.cc:2781] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.838343 2067 raft_consensus.cc:686] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.838672 2067 consensus_queue.cc:237] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.838909 32133 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 7.
I20240501 08:14:42.839279 32132 raft_consensus.cc:2396] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:42.839741 31938 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.840139 2074 raft_consensus.cc:2781] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:42.840411 2074 raft_consensus.cc:686] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.840744 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.840866 1777 raft_consensus.cc:686] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.840452 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.841120 32000 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.841233 32000 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.841272 1777 consensus_queue.cc:237] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.840952 31872 raft_consensus.cc:3032] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.842015 31872 raft_consensus.cc:729] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.840798 2074 consensus_queue.cc:237] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.842329 31872 consensus_queue.cc:260] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.842643 31872 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.844534 32000 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.845221 32000 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.845702 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.845969 32261 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.847754 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:42.848008 32132 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.849349 32261 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.849900 31938 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:42.850219 1777 raft_consensus.cc:2781] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.850431 1777 raft_consensus.cc:686] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.850935 1777 consensus_queue.cc:237] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.851343 32132 raft_consensus.cc:2396] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:42.851109 31731 catalog_manager.cc:5484] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 7 to 8, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.851644 31731 catalog_manager.cc:5484] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 7 to 8, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:42.852416 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:42.852842 31872 raft_consensus.cc:2396] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20240501 08:14:42.853209 32000 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.853319 32000 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.856035 32000 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.856638 32000 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.857087 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.857172 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.857411 32261 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.857450 31869 raft_consensus.cc:3032] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.857539 31869 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.857894 31869 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.858263 31869 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.861119 31869 raft_consensus.cc:2396] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:42.863299 32261 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 8.
I20240501 08:14:42.863824 31938 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:42.863898 31731 catalog_manager.cc:5484] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 8, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.864118 1777 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.864435 1777 raft_consensus.cc:686] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.864431 31731 catalog_manager.cc:5484] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 8, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.864838 1777 consensus_queue.cc:237] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.864871 31731 catalog_manager.cc:5484] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 6 to 7, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 7 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:42.870544 31731 catalog_manager.cc:5484] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 7 to 8, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 8 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.875831 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.876045 32132 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.876156 32132 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.878121 32132 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.878679 32132 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.879182 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.879283 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.879441 31869 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.879520 32261 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.879869 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.880038 32132 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.880144 32132 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.881412 31869 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:42.881822 32069 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.882158 1945 raft_consensus.cc:2781] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.882287 1945 raft_consensus.cc:686] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.882412 32261 raft_consensus.cc:2396] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:42.882517 1945 consensus_queue.cc:237] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.882990 32132 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.883515 32132 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.884115 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.884205 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.884341 32000 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:42.884461 32000 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.884510 31869 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.884721 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.884776 32000 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.884917 32133 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.885035 32133 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.885097 32000 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.887926 32000 raft_consensus.cc:2396] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240501 08:14:42.887874 32133 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.888823 32133 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.888929 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.889102 32000 raft_consensus.cc:3032] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.889212 32000 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.889341 31869 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 9.
I20240501 08:14:42.889489 32000 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.889808 32000 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.889909 32069 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:42.890241 1245 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:42.889904 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93a196489f264058a064dc84690ba947"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.891507 32133 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.891621 32133 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.891893 32000 raft_consensus.cc:2396] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:42.892269 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.892527 32261 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.894918 32261 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:42.895918 32068 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:42.896234 32133 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.897475 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.897774 2089 raft_consensus.cc:2781] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.897873 31869 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.897936 2089 raft_consensus.cc:686] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.898221 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.898449 32000 raft_consensus.cc:3032] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:42.898314 2089 consensus_queue.cc:237] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.898576 32000 raft_consensus.cc:729] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.898996 32133 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.898962 32000 consensus_queue.cc:260] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.899559 32000 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.899976 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.900240 32132 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.900418 32132 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.903107 32000 raft_consensus.cc:2396] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:42.903617 31869 raft_consensus.cc:2396] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:42.904255 32132 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.904870 32068 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:42.905148 2089 raft_consensus.cc:2726] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 08:14:42.905205 32132 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.905747 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.905952 31869 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.906323 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.906488 32132 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.906591 32132 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.906584 31731 catalog_manager.cc:5484] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 6 to 8, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:42.908037 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.908612 32000 raft_consensus.cc:3032] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.908906 31869 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:42.909312 32069 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.909953 1245 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.909988 2089 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.910348 2089 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.910266 1245 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.910982 32132 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.911571 32132 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.911499 2089 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.912447 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.912508 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.912662 32002 raft_consensus.cc:3032] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.912747 32261 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.912760 32002 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.913026 32002 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.913292 32002 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.908715 32000 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.913944 32000 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.914261 32000 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.916003 32261 raft_consensus.cc:2454] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:42.916496 32068 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:42.917091 32002 raft_consensus.cc:2396] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:42.917428 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.917625 32132 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.917735 32132 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.918764 32000 raft_consensus.cc:2396] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20240501 08:14:42.918707 31731 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 9 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:42.919243 31731 catalog_manager.cc:5484] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 7 to 8, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.919660 31731 catalog_manager.cc:5484] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 reported cstate change: term changed from 7 to 8, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.920396 32132 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.922255 32132 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.922982 2089 raft_consensus.cc:2781] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.923223 2089 raft_consensus.cc:686] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.923482 2089 consensus_queue.cc:237] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.924098 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.924502 32132 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.924515 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.924746 32132 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.925160 32261 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.927237 31728 catalog_manager.cc:5484] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 7 to 8, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.928824 32261 raft_consensus.cc:2454] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:42.929317 32132 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.930181 32132 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.930270 32068 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:42.930514 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.930943 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.931258 32261 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.931509 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.931761 2089 raft_consensus.cc:2781] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.931798 32132 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.931919 2089 raft_consensus.cc:686] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.931943 32132 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.932201 2089 consensus_queue.cc:237] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.933451 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.933645 32000 raft_consensus.cc:3032] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.933741 32000 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.935161 31731 catalog_manager.cc:5484] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 7 to 8, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:42.935987 32261 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 9.
I20240501 08:14:42.935930 32132 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.937088 32068 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:42.937139 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.937422 31869 raft_consensus.cc:3032] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:42.937538 31869 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:42.937808 2089 raft_consensus.cc:2781] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:42.937985 2089 raft_consensus.cc:686] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.937916 31869 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.938231 2089 consensus_queue.cc:237] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.938313 31869 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.933995 32000 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:42.934056 32002 raft_consensus.cc:3032] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:42.940145 32002 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.940459 32002 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.940661 32002 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.942520 32002 raft_consensus.cc:2396] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 08:14:42.940042 32000 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.944458 31869 raft_consensus.cc:2396] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:42.945524 32132 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 7 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.945825 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.946130 32261 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:42.946444 32000 raft_consensus.cc:2396] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:42.946776 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.946996 32133 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.947124 32133 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.945333 31731 catalog_manager.cc:5484] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 9 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.949811 32261 raft_consensus.cc:2396] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:42.950356 32068 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:42.950946 2089 raft_consensus.cc:2726] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20240501 08:14:42.952128 32133 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.953229 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.953490 31869 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.955220 32133 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:42.956354 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.956553 32132 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.956486 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.956696 32132 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.956737 32000 raft_consensus.cc:3032] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:42.956853 32000 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.957250 32000 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.956211 31869 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 9.
I20240501 08:14:42.958343 32000 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.958645 32069 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.959369 32132 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.960126 32132 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.960407 2089 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:42.960584 2089 raft_consensus.cc:686] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.960541 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.960748 32133 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.960769 32002 raft_consensus.cc:3032] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:42.960852 32002 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.960896 2089 consensus_queue.cc:237] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:42.961130 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.961124 32002 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.961319 31869 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.961412 32002 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:42.962500 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.962675 32133 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.962781 32133 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.963514 31869 raft_consensus.cc:2396] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:42.964166 32002 raft_consensus.cc:2396] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:42.964715 32069 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:42.964984 1245 raft_consensus.cc:2726] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 08:14:42.965523 32133 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.966176 32133 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:42.966097 1996 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.966964 32000 raft_consensus.cc:2396] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 08:14:42.966792 31727 catalog_manager.cc:5484] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 9 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:42.967154 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.967392 32261 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.969238 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f78273f47414fd199e593595e6aae23"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.969436 32133 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.969532 32133 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.970425 32261 raft_consensus.cc:2396] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:42.971208 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.971438 31869 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.974047 32133 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.974256 32261 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Refusing update from remote peer d6949d828ece45dcb729633be6d49c97: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20240501 08:14:42.974702 32133 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.974793 2048 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:42.975780 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.976012 32133 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.976150 32133 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.977771 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:42.978025 32002 raft_consensus.cc:3032] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:42.978026 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.978123 32002 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:42.978215 32261 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.978420 32002 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.978734 32002 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.979282 32133 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.979590 31869 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:42.980460 32069 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:42.980783 1245 raft_consensus.cc:2781] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.980927 1245 raft_consensus.cc:686] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.980945 32261 raft_consensus.cc:2454] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:42.981194 1245 consensus_queue.cc:237] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:42.981345 32068 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:42.981719 2083 raft_consensus.cc:2781] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.981858 2083 raft_consensus.cc:686] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.982142 2083 consensus_queue.cc:237] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.982594 32002 raft_consensus.cc:2396] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:42.983716 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:42.983983 31869 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.988309 31869 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:42.988924 32069 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:42.989378 1945 raft_consensus.cc:2781] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:42.989554 1945 raft_consensus.cc:686] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:42.989931 1945 consensus_queue.cc:237] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:42.995043 31728 catalog_manager.cc:5484] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 reported cstate change: term changed from 6 to 8, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:42.995733 31728 catalog_manager.cc:5484] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 7 to 8, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:42.996470 32133 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:42.996850 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:42.997160 32261 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:42.998553 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:42.998870 32133 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:42.999048 32133 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.001675 32133 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.002002 32261 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:43.002748 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.003011 32263 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.004889 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.005105 31869 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.005869 32263 raft_consensus.cc:2396] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:43.008311 31869 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:43.008219 31727 catalog_manager.cc:5484] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 reported cstate change: term changed from 6 to 8, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.008867 32069 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:43.009456 1945 raft_consensus.cc:2781] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:43.009604 1945 raft_consensus.cc:686] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.009912 1945 consensus_queue.cc:237] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.011235 32133 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.012483 32132 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:43.012709 32132 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.012827 32132 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:43.015094 32132 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.015729 32132 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 7 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.016182 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.016480 31872 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:43.020162 31872 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 7.
I20240501 08:14:43.021095 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e872bea48fc479686691121187aa0db"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:43.021203 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.021301 32133 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.021423 32133 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.021428 32002 raft_consensus.cc:3032] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:43.021510 32002 raft_consensus.cc:729] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.021792 32002 consensus_queue.cc:260] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.022130 32002 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:43.024401 32133 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.026350 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.026618 32000 raft_consensus.cc:3032] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.026721 32000 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.026734 32133 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.026799 31731 catalog_manager.cc:5484] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 6 to 8, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.026997 32000 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.027314 32000 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.028254 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.028949 32002 raft_consensus.cc:2396] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:43.029186 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:43.029428 32133 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.029537 32133 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.030040 32000 raft_consensus.cc:2396] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 08:14:43.031543 32263 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.032423 32133 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.033073 32133 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.033819 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.033834 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.034032 32000 raft_consensus.cc:3032] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.034052 32261 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.034029 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:43.034123 32000 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.034209 32133 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.034318 32133 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:43.034389 32000 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.034682 32000 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.037062 32069 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.037341 1945 raft_consensus.cc:2781] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:43.037392 31869 raft_consensus.cc:1261] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 08:14:43.037493 1945 raft_consensus.cc:686] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 7 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.037525 32261 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 9.
I20240501 08:14:43.037751 1945 consensus_queue.cc:237] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.038049 1777 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.038552 32068 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:43.039112 2083 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.039301 2083 raft_consensus.cc:686] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.039639 2083 consensus_queue.cc:237] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.041057 32000 raft_consensus.cc:2396] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 08:14:43.042397 32133 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.043459 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.043684 31869 raft_consensus.cc:3032] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:43.043769 31869 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.044025 31869 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.044412 31869 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:43.045495 32261 raft_consensus.cc:1261] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 08:14:43.046097 2067 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.047956 31869 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 7.
I20240501 08:14:43.048431 32069 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.048796 2083 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:43.048936 2083 raft_consensus.cc:686] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 7 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.049204 2083 consensus_queue.cc:237] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.052565 32133 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 7 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.053148 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.053382 32261 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:43.055665 32261 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 7.
I20240501 08:14:43.057725 32263 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 9.
I20240501 08:14:43.058102 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:43.058166 32068 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.058456 32133 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.058631 32133 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.058706 1945 raft_consensus.cc:2781] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.058501 31723 catalog_manager.cc:5484] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 reported cstate change: term changed from 6 to 7, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 7 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.059047 1945 raft_consensus.cc:686] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.059007 31723 catalog_manager.cc:5484] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 9 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.059379 1945 consensus_queue.cc:237] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.062469 32133 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.063902 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.064198 31869 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.067032 31869 raft_consensus.cc:2396] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:43.067755 32133 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.069001 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:43.069183 32133 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.069291 32133 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.072345 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.072572 32000 raft_consensus.cc:3032] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.072686 32000 raft_consensus.cc:729] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.073040 32133 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.074116 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.074324 31869 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.074210 32000 consensus_queue.cc:260] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.074522 32000 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.077189 31869 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 8.
I20240501 08:14:43.077670 32069 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.078231 1945 raft_consensus.cc:2781] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:43.078486 1945 raft_consensus.cc:686] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.078752 32133 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.079478 32263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.079720 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:43.079681 32263 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.079941 32133 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.080051 32133 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.078752 1945 consensus_queue.cc:237] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.082957 32133 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.082850 31723 catalog_manager.cc:5484] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 reported cstate change: term changed from 6 to 7, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 7 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.083408 31723 catalog_manager.cc:5484] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 9 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.083998 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.084167 32133 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.084250 32002 raft_consensus.cc:3032] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.084364 32002 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.084872 32261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.085101 32261 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.085528 32002 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.085894 32002 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.086098 32133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5"
dest_uuid: "4827555155084237b0199283e5896782"
from {username='slave'} at 127.0.0.1:52872
I20240501 08:14:43.086277 32133 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.086405 32133 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.089267 32133 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.090415 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.090673 31869 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.091768 32002 raft_consensus.cc:2396] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 08:14:43.093354 32261 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4827555155084237b0199283e5896782 in term 9.
I20240501 08:14:43.093502 31869 raft_consensus.cc:2396] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:43.093955 32068 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.094224 32133 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.094725 2089 raft_consensus.cc:2781] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.094920 2089 raft_consensus.cc:686] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Leader. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.095274 2089 consensus_queue.cc:237] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.095621 32000 raft_consensus.cc:2396] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240501 08:14:43.096982 32068 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:43.097239 1245 raft_consensus.cc:2726] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 08:14:43.097915 32263 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:43.098306 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "4827555155084237b0199283e5896782" candidate_term: 8 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.098486 32000 raft_consensus.cc:3032] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:43.098585 32000 raft_consensus.cc:729] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.098847 32000 consensus_queue.cc:260] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.099083 32000 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.101223 32000 raft_consensus.cc:2396] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4827555155084237b0199283e5896782 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240501 08:14:43.101687 32068 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4827555155084237b0199283e5896782; no voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:43.102080 2089 raft_consensus.cc:2726] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240501 08:14:43.113852 31723 catalog_manager.cc:5484] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 reported cstate change: term changed from 6 to 8, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 8 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.119686 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.119946 32263 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.120072 32263 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.121908 31727 catalog_manager.cc:5484] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 4827555155084237b0199283e5896782 (127.6.56.67). New cstate: current_term: 9 leader_uuid: "4827555155084237b0199283e5896782" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.123570 32263 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.124516 32263 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.125950 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.126216 31869 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.129937 31869 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.130558 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.130630 32199 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.130851 32263 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.130998 32263 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.131078 1843 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.131258 1843 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.131644 1843 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.136386 32263 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.145591 32263 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.145720 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.146056 31869 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.147346 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.147578 32261 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.147709 32261 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.149118 31869 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.149747 32199 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.150203 1978 raft_consensus.cc:2781] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.150391 1978 raft_consensus.cc:686] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.150525 32261 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.150774 1978 consensus_queue.cc:237] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.151127 32261 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.152120 32261 raft_consensus.cc:1261] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 08:14:43.153076 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.153240 32261 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.153357 32261 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.154714 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.154924 32000 raft_consensus.cc:3032] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.155014 32000 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.155267 32000 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.155632 32000 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.160265 31727 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 9 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.164314 2067 consensus_queue.cc:1035] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:43.166674 32133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.166913 32133 raft_consensus.cc:3032] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.167013 32133 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.167323 32133 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.167632 32133 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.165433 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.168277 32113 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240501 08:14:43.168434 32002 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.169924 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.170104 32132 raft_consensus.cc:3032] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.170207 32132 raft_consensus.cc:729] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.170460 32132 consensus_queue.cc:260] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.170750 32132 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.171310 32002 raft_consensus.cc:2396] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.173552 32133 raft_consensus.cc:2396] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 08:14:43.175410 32132 raft_consensus.cc:2396] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.182327 32261 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.187177 32261 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.187734 32000 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.190505 31869 raft_consensus.cc:1261] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 08:14:43.191176 2067 consensus_queue.cc:1035] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:43.191377 2067 consensus_queue.cc:1225] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Peer d6949d828ece45dcb729633be6d49c97 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 08:14:43.192202 31869 pending_rounds.cc:76] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 4
I20240501 08:14:43.192332 31869 pending_rounds.cc:97] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 08:14:43.192405 31869 raft_consensus.cc:2866] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.217538 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.218505 1304 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:43.218281 32261 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.219406 32261 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.220217 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.220381 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.220508 32130 raft_consensus.cc:3032] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.220588 32130 raft_consensus.cc:729] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.220641 32000 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.220894 32130 consensus_queue.cc:260] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.221202 32130 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.224184 32000 raft_consensus.cc:2396] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.224557 32130 raft_consensus.cc:2396] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.232896 32198 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.233475 1978 raft_consensus.cc:2726] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 08:14:43.233693 32198 leader_election.cc:302] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.233951 1978 raft_consensus.cc:2726] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 08:14:43.235963 32261 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.236377 32263 raft_consensus.cc:1226] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Rejecting Update request from peer 6b7faa87de354980a76c80fb864f9edf for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20240501 08:14:43.236750 32261 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 8 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.238775 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.238986 31869 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.241788 31869 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 8.
I20240501 08:14:43.242267 32199 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.242568 1978 raft_consensus.cc:2781] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:43.242720 1978 raft_consensus.cc:686] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.242992 1978 consensus_queue.cc:237] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.243357 31869 raft_consensus.cc:1226] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Rejecting Update request from peer 6b7faa87de354980a76c80fb864f9edf for earlier term 7. Current term is 8. Ops: []
I20240501 08:14:43.243927 1777 consensus_queue.cc:1046] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:43.244148 1777 consensus_queue.cc:1225] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Peer d6949d828ece45dcb729633be6d49c97 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 08:14:43.244278 1777 raft_consensus.cc:3032] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:43.244418 1777 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.244642 1777 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.244983 1777 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.249109 31727 catalog_manager.cc:5484] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 8 to 9, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 9 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.252808 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.253202 32000 raft_consensus.cc:2396] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.255620 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.255810 32261 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.255929 32261 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:43.260052 31869 raft_consensus.cc:1261] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 08:14:43.260664 31727 catalog_manager.cc:5484] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 7 to 8, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 8 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.260972 1696 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:43.261690 1566 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:43.264204 32263 raft_consensus.cc:1261] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 08:14:43.264590 31852 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:43.264832 2074 consensus_queue.cc:1035] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:43.276815 31869 raft_consensus.cc:1261] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20240501 08:14:43.276965 31869 pending_rounds.cc:76] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 6
I20240501 08:14:43.277032 31869 pending_rounds.cc:97] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20240501 08:14:43.277112 31869 raft_consensus.cc:2866] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.277652 1777 consensus_queue.cc:1035] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.298048 31869 raft_consensus.cc:1261] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20240501 08:14:43.298197 31869 pending_rounds.cc:76] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 6
I20240501 08:14:43.298262 31869 pending_rounds.cc:97] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20240501 08:14:43.298348 31869 raft_consensus.cc:2866] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.298977 1777 consensus_queue.cc:1035] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.313040 1777 consensus_queue.cc:1035] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.316220 32260 raft_consensus.cc:1261] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20240501 08:14:43.317946 32260 raft_consensus.cc:1261] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 08:14:43.319027 32260 raft_consensus.cc:1261] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Refusing update from remote peer 6b7faa87de354980a76c80fb864f9edf: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 08:14:43.333768 32260 raft_consensus.cc:1261] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240501 08:14:43.334447 1945 consensus_queue.cc:1035] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.340204 32000 raft_consensus.cc:1261] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20240501 08:14:43.340356 32000 pending_rounds.cc:76] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 5
I20240501 08:14:43.340409 32000 pending_rounds.cc:97] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20240501 08:14:43.340479 32000 raft_consensus.cc:2866] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.341421 2089 consensus_queue.cc:1035] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.341627 2089 consensus_queue.cc:1225] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [LEADER]: Peer 6b7faa87de354980a76c80fb864f9edf log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:43.342505 32000 pending_rounds.cc:76] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 3
I20240501 08:14:43.342607 32000 pending_rounds.cc:97] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:43.342684 32000 raft_consensus.cc:2866] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.342772 32000 pending_rounds.cc:97] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240501 08:14:43.342820 32000 raft_consensus.cc:2866] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.343698 1436 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 08:14:43.345602 32261 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.346846 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.347019 32263 raft_consensus.cc:1226] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Rejecting Update request from peer 6b7faa87de354980a76c80fb864f9edf for earlier term 6. Current term is 7. Ops: []
I20240501 08:14:43.347059 32000 raft_consensus.cc:3032] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 6 LEADER]: Stepping down as leader of term 6
I20240501 08:14:43.347157 32000 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.347541 1777 consensus_queue.cc:1035] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 08:14:43.347466 32000 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.347904 32000 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:43.350538 32261 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.351476 32000 raft_consensus.cc:2396] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 08:14:43.351641 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.351866 32261 raft_consensus.cc:482] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.351996 32261 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.352022 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.352257 31869 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 6 FOLLOWER]: Advancing to term 7
I20240501 08:14:43.352875 32242 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 08:14:43.354911 32261 raft_consensus.cc:504] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.355053 31869 raft_consensus.cc:2454] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 7.
I20240501 08:14:43.355576 32199 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.355753 32261 leader_election.cc:288] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.356096 2131 raft_consensus.cc:2781] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Leader election won for term 7
I20240501 08:14:43.356427 2131 raft_consensus.cc:686] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 7 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.356406 1777 consensus_queue.cc:1035] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.356474 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.356702 32130 raft_consensus.cc:3032] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.356793 32130 raft_consensus.cc:729] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.357074 32130 consensus_queue.cc:260] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.357018 2131 consensus_queue.cc:237] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.357316 1777 consensus_queue.cc:1035] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.357409 32130 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.359542 32130 raft_consensus.cc:2396] T d816475cd23d4cf1967ef9a5f6a1ac18 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.360960 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f78273f47414fd199e593595e6aae23"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.361129 32261 raft_consensus.cc:482] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.361246 32261 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.361567 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d816475cd23d4cf1967ef9a5f6a1ac18" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.361862 32000 raft_consensus.cc:3037] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.362500 31727 catalog_manager.cc:5484] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 6 to 7, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 7 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.363879 32263 raft_consensus.cc:1261] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 08:14:43.365906 2089 consensus_queue.cc:1035] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.366129 2089 consensus_queue.cc:1225] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [LEADER]: Peer 64f7786244bd4f90af1ad476966482c4 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:43.366436 32000 raft_consensus.cc:2396] T d816475cd23d4cf1967ef9a5f6a1ac18 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.367154 32198 leader_election.cc:302] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.367744 2131 raft_consensus.cc:2726] T d816475cd23d4cf1967ef9a5f6a1ac18 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 08:14:43.377485 32262 pending_rounds.cc:76] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4: Aborting all ops after (but not including) 3
I20240501 08:14:43.377704 32262 pending_rounds.cc:97] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20240501 08:14:43.377875 32262 raft_consensus.cc:2866] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.378010 32262 pending_rounds.cc:97] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20240501 08:14:43.378131 32262 raft_consensus.cc:2866] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.382529 31869 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20240501 08:14:43.383489 1945 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:43.390619 31869 raft_consensus.cc:1261] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240501 08:14:43.391466 31983 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 08:14:43.391386 2089 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:43.396732 32001 raft_consensus.cc:1261] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20240501 08:14:43.396898 32001 pending_rounds.cc:76] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 7
I20240501 08:14:43.396976 32001 pending_rounds.cc:97] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20240501 08:14:43.397068 32001 raft_consensus.cc:2866] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.397864 1245 consensus_queue.cc:1035] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:43.398097 32261 raft_consensus.cc:504] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.399260 32003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.399504 32003 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.399544 32261 leader_election.cc:288] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.400012 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f78273f47414fd199e593595e6aae23" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.400214 32130 raft_consensus.cc:3032] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.400285 32130 raft_consensus.cc:729] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.400543 32130 consensus_queue.cc:260] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.400825 32130 raft_consensus.cc:3037] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.403224 32003 raft_consensus.cc:2396] T 4f78273f47414fd199e593595e6aae23 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.403813 32130 raft_consensus.cc:2396] T 4f78273f47414fd199e593595e6aae23 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.404268 32198 leader_election.cc:302] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.404919 2131 raft_consensus.cc:2726] T 4f78273f47414fd199e593595e6aae23 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 08:14:43.412515 32261 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.412819 32261 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.412967 32261 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.416045 32000 raft_consensus.cc:1261] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20240501 08:14:43.416216 32000 pending_rounds.cc:76] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 7
I20240501 08:14:43.416283 32000 pending_rounds.cc:97] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20240501 08:14:43.416404 32000 raft_consensus.cc:2866] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.421038 1945 consensus_queue.cc:1035] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:43.426365 32261 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.427583 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.427790 31869 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.430717 31869 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 8.
I20240501 08:14:43.431301 32199 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.431792 2132 raft_consensus.cc:2781] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:43.432086 2132 raft_consensus.cc:686] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.432507 2132 consensus_queue.cc:237] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.434487 31869 raft_consensus.cc:1261] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 08:14:43.435076 1945 consensus_queue.cc:1035] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.445003 32261 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 8 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.445660 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.445901 32000 raft_consensus.cc:3032] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:43.445989 32000 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.446257 32000 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.446604 32000 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.448240 32262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.448470 32262 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.448580 32262 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.449489 32002 raft_consensus.cc:1261] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20240501 08:14:43.449594 32002 pending_rounds.cc:76] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 5
I20240501 08:14:43.449678 32002 pending_rounds.cc:97] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20240501 08:14:43.449748 32002 raft_consensus.cc:2866] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.450258 32001 raft_consensus.cc:1261] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20240501 08:14:43.450387 32001 pending_rounds.cc:76] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 4
I20240501 08:14:43.450446 32001 pending_rounds.cc:97] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 08:14:43.450520 32001 raft_consensus.cc:2866] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.450608 32001 pending_rounds.cc:97] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20240501 08:14:43.450685 32001 raft_consensus.cc:2866] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.451850 32262 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.452839 2089 consensus_queue.cc:1035] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.453109 2089 consensus_queue.cc:1225] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [LEADER]: Peer 6b7faa87de354980a76c80fb864f9edf log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:43.453480 2089 consensus_queue.cc:1035] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:43.453954 32000 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 8.
I20240501 08:14:43.454864 31868 raft_consensus.cc:1261] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 08:14:43.455080 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.455299 32130 raft_consensus.cc:3032] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.455307 32262 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.455384 32130 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.455657 32130 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.455938 32130 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.456519 31868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.456761 31868 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.458781 32130 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 08:14:43.459761 31868 raft_consensus.cc:2454] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.460618 32262 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.460816 32199 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:43.460898 32262 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.461023 32262 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.461301 2131 raft_consensus.cc:2781] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.461464 2131 raft_consensus.cc:686] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.461742 2131 consensus_queue.cc:237] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.463840 32262 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.464541 32000 pending_rounds.cc:76] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 3
I20240501 08:14:43.464671 32000 pending_rounds.cc:97] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:43.464771 32000 raft_consensus.cc:2866] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.464905 32000 pending_rounds.cc:97] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240501 08:14:43.464995 32000 raft_consensus.cc:2866] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.470124 32263 raft_consensus.cc:1261] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240501 08:14:43.469810 31868 raft_consensus.cc:1261] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20240501 08:14:43.470577 32001 raft_consensus.cc:1261] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20240501 08:14:43.470723 32001 pending_rounds.cc:76] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 6
I20240501 08:14:43.470795 32001 pending_rounds.cc:97] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20240501 08:14:43.470876 32001 raft_consensus.cc:2866] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.470949 2083 consensus_queue.cc:1035] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:43.471189 32262 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.472083 31870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.472311 31870 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.472416 2089 consensus_queue.cc:1035] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.475173 31870 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.475172 2089 consensus_queue.cc:1035] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.477027 32263 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.477173 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.477238 32263 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.477366 32263 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.477360 32000 raft_consensus.cc:3032] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.477458 32000 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.477747 32000 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.478139 32000 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.480278 32263 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.482398 32263 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.481287 32199 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.483021 1978 raft_consensus.cc:2781] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.483167 1978 raft_consensus.cc:686] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.483282 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.483484 32130 raft_consensus.cc:3032] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.483402 31870 raft_consensus.cc:1261] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20240501 08:14:43.483558 32130 raft_consensus.cc:729] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.483445 1978 consensus_queue.cc:237] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.483800 32130 consensus_queue.cc:260] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.484064 32130 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.484275 2083 consensus_queue.cc:1035] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.485472 31870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.487042 32130 raft_consensus.cc:2396] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 08:14:43.487133 32259 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e872bea48fc479686691121187aa0db"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.487676 32198 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97
I20240501 08:14:43.488133 1843 raft_consensus.cc:2726] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20240501 08:14:43.490120 31869 raft_consensus.cc:1261] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20240501 08:14:43.491053 2089 consensus_queue.cc:1035] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.494554 32000 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.495334 32263 raft_consensus.cc:1261] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240501 08:14:43.496120 2089 consensus_queue.cc:1035] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.503561 32258 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 08:14:43.504308 2089 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.507997 31869 raft_consensus.cc:1261] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20240501 08:14:43.508846 2083 consensus_queue.cc:1035] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.512588 32259 raft_consensus.cc:482] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.512791 32259 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.515969 32259 raft_consensus.cc:504] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.518060 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.518303 32130 raft_consensus.cc:3032] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.518404 32130 raft_consensus.cc:729] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.518704 32130 consensus_queue.cc:260] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.519070 32130 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.521934 32130 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 10.
I20240501 08:14:43.522397 32198 leader_election.cc:302] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:43.522771 32259 leader_election.cc:288] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 10 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.523157 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e872bea48fc479686691121187aa0db" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.522804 2132 raft_consensus.cc:2781] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.523680 2132 raft_consensus.cc:686] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [term 10 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.524433 32000 raft_consensus.cc:3037] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.524395 32258 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.524137 2132 consensus_queue.cc:237] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.527635 32000 raft_consensus.cc:2454] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 10.
I20240501 08:14:43.529033 32001 raft_consensus.cc:1261] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240501 08:14:43.531455 32258 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.531594 32258 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.531639 2132 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240501 08:14:43.534515 32001 raft_consensus.cc:1261] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Refusing update from remote peer 4827555155084237b0199283e5896782: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20240501 08:14:43.534606 32001 pending_rounds.cc:76] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 5
I20240501 08:14:43.534700 32001 pending_rounds.cc:97] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20240501 08:14:43.534754 32001 raft_consensus.cc:2866] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.534984 32258 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.535305 1945 consensus_queue.cc:1035] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.535496 1945 consensus_queue.cc:1225] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [LEADER]: Peer 6b7faa87de354980a76c80fb864f9edf log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:43.536549 32001 pending_rounds.cc:76] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 3
I20240501 08:14:43.536624 32001 pending_rounds.cc:97] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:43.536705 32001 raft_consensus.cc:2866] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.536777 32001 pending_rounds.cc:97] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240501 08:14:43.536840 32001 raft_consensus.cc:2866] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.537671 32258 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.537719 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.537947 31872 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.538784 32260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.538972 32260 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.539079 32260 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.539045 32001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.539227 32001 raft_consensus.cc:3032] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.539292 32001 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.539522 32001 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.539850 32001 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.540712 31872 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.541148 32199 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.541695 2132 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.541935 2132 raft_consensus.cc:686] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.542152 2132 consensus_queue.cc:237] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.542819 32260 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.543509 32260 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.543993 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.544169 32002 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.544504 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.544710 32130 raft_consensus.cc:3032] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.544788 32130 raft_consensus.cc:729] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.545064 32130 consensus_queue.cc:260] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.545315 32130 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.546387 32001 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.546137 31727 catalog_manager.cc:5484] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 7 to 8, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 8 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.547086 32002 raft_consensus.cc:2396] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 08:14:43.547320 32260 raft_consensus.cc:1226] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Rejecting Update request from peer 4827555155084237b0199283e5896782 for earlier term 9. Current term is 10. Ops: []
I20240501 08:14:43.548503 32260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.548736 32260 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.548856 32260 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.549685 32130 raft_consensus.cc:2396] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 08:14:43.550052 32198 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4; no voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.550534 2131 raft_consensus.cc:2726] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240501 08:14:43.552886 31872 raft_consensus.cc:1261] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20240501 08:14:43.552098 32260 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.553468 2131 consensus_queue.cc:1035] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 08:14:43.553673 32260 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 8 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.554118 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.554313 31872 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.556166 32260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.556360 32260 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.556473 32260 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.556546 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.556775 32130 raft_consensus.cc:3032] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:43.556864 32130 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.556969 31872 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 8.
I20240501 08:14:43.557305 32130 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.557420 32199 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.557688 32130 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.557698 1843 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:43.557842 1843 raft_consensus.cc:686] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.558094 1843 consensus_queue.cc:237] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.558514 31727 catalog_manager.cc:5484] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 9 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.559044 31727 catalog_manager.cc:5484] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 8 to 9, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 9 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.559239 32260 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.559465 31727 catalog_manager.cc:5484] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 9 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.559888 32260 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.560369 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.560551 32132 raft_consensus.cc:3032] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.560662 32132 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.560765 32130 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 8.
I20240501 08:14:43.560920 32132 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.561024 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.561187 32132 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.561223 31872 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.563691 32132 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 08:14:43.564438 31727 catalog_manager.cc:5484] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 9 to 10, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 10 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.567221 31872 raft_consensus.cc:2454] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.567636 32199 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:43.568059 2131 raft_consensus.cc:2781] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.568265 2131 raft_consensus.cc:686] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.568562 2131 consensus_queue.cc:237] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.570453 32260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.570660 32260 raft_consensus.cc:482] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.570772 32260 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.573320 31723 catalog_manager.cc:5484] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 7 to 8, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 8 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.573737 31723 catalog_manager.cc:5484] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 8 to 9, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 9 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.573753 32260 raft_consensus.cc:504] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.574410 32260 leader_election.cc:288] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.575259 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.575273 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9fa92579c134968bc91552c0542b1fc" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.575475 32132 raft_consensus.cc:3032] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.575491 32000 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.575569 32132 raft_consensus.cc:729] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.575834 32132 consensus_queue.cc:260] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.576138 32132 raft_consensus.cc:3037] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.578205 32132 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 10.
I20240501 08:14:43.578454 32000 raft_consensus.cc:2454] T b9fa92579c134968bc91552c0542b1fc P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 10.
I20240501 08:14:43.578727 32198 leader_election.cc:302] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters:
I20240501 08:14:43.579042 2131 raft_consensus.cc:2781] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.579339 2131 raft_consensus.cc:686] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [term 10 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.579308 32260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.579627 32260 raft_consensus.cc:482] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.579789 32260 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.579758 2131 consensus_queue.cc:237] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.582756 32260 raft_consensus.cc:504] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.582921 31723 catalog_manager.cc:5484] T b9fa92579c134968bc91552c0542b1fc P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 9 to 10, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 10 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.583459 32260 leader_election.cc:288] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.584388 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.584614 32000 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.584657 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84680e97e91b4e16906adc6a61d2ef9a" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.584975 32132 raft_consensus.cc:3032] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.585520 32132 raft_consensus.cc:729] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.585901 32132 consensus_queue.cc:260] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.586141 32260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.586388 32260 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.586494 32260 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.587170 32132 raft_consensus.cc:3037] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.587383 32000 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 10.
I20240501 08:14:43.587894 32198 leader_election.cc:302] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:43.588223 2131 raft_consensus.cc:2781] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.588477 2131 raft_consensus.cc:686] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [term 10 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.588768 2131 consensus_queue.cc:237] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.589481 32260 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.590355 32132 raft_consensus.cc:2454] T 84680e97e91b4e16906adc6a61d2ef9a P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 10.
I20240501 08:14:43.590822 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.591069 31872 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.594041 31872 raft_consensus.cc:2396] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 64f7786244bd4f90af1ad476966482c4 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20240501 08:14:43.593717 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.594436 32130 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.594590 32260 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 8 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.595999 32260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.596200 32260 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.596378 32260 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.597422 32130 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 8.
I20240501 08:14:43.597841 32198 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:43.597720 31723 catalog_manager.cc:5484] T 84680e97e91b4e16906adc6a61d2ef9a P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 9 to 10, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 10 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.598264 2131 raft_consensus.cc:2781] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:43.598475 2131 raft_consensus.cc:686] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.598732 2131 consensus_queue.cc:237] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.599224 32260 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.600281 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.600522 31872 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.603988 31872 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.604822 32199 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.605221 2132 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.605396 2132 raft_consensus.cc:686] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.605772 2132 consensus_queue.cc:237] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.608001 32260 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.608775 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.609092 32000 raft_consensus.cc:3032] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.609189 32000 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.609331 32260 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd"
dest_uuid: "64f7786244bd4f90af1ad476966482c4"
from {username='slave'} at 127.0.0.1:44504
I20240501 08:14:43.609577 32260 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.609706 32260 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.609578 32000 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.610062 32000 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.613962 32260 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.614305 32000 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.615790 31872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.615849 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "64f7786244bd4f90af1ad476966482c4" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.616122 31872 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.616129 32130 raft_consensus.cc:3032] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.616258 32130 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.616806 32130 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.617329 32130 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.620959 31872 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.621601 32130 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64f7786244bd4f90af1ad476966482c4 in term 9.
I20240501 08:14:43.621703 32199 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.622128 1978 raft_consensus.cc:2781] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.622323 1978 raft_consensus.cc:686] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Leader. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.622669 1978 consensus_queue.cc:237] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.626036 31869 raft_consensus.cc:1261] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20240501 08:14:43.626940 2132 consensus_queue.cc:1035] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.629555 32260 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [CANDIDATE]: Term 9 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.632547 31727 catalog_manager.cc:5484] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 6 to 8, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 8 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.636641 32130 raft_consensus.cc:1261] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20240501 08:14:43.636768 32130 pending_rounds.cc:76] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 5
I20240501 08:14:43.636848 32130 pending_rounds.cc:97] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20240501 08:14:43.636912 32130 raft_consensus.cc:2866] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.637975 2132 consensus_queue.cc:1035] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 08:14:43.638240 2132 consensus_queue.cc:1225] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer 4827555155084237b0199283e5896782 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20240501 08:14:43.639421 32130 pending_rounds.cc:76] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: Aborting all ops after (but not including) 3
I20240501 08:14:43.639515 32130 pending_rounds.cc:97] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240501 08:14:43.639597 32130 raft_consensus.cc:2866] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.639686 32130 pending_rounds.cc:97] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20240501 08:14:43.639763 32130 raft_consensus.cc:2866] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.640157 31727 catalog_manager.cc:5484] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 8 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 9 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.640810 31727 catalog_manager.cc:5484] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 reported cstate change: term changed from 8 to 9, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 64f7786244bd4f90af1ad476966482c4 (127.6.56.68). New cstate: current_term: 9 leader_uuid: "64f7786244bd4f90af1ad476966482c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.659998 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.660243 31869 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.660374 31869 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.663646 31869 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.664778 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.665014 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.665097 32130 raft_consensus.cc:3032] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.665210 32130 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.665232 32000 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.665566 32130 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.666051 32130 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.667986 32000 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 9.
I20240501 08:14:43.668532 31807 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.668921 2148 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.669467 2148 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 9 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.669811 2148 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.672825 31727 catalog_manager.cc:5484] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 8 to 9, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 9 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.673277 32130 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 9.
I20240501 08:14:43.676889 31869 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.677410 31872 raft_consensus.cc:1261] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20240501 08:14:43.678006 1978 consensus_queue.cc:1035] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:43.678218 1978 consensus_queue.cc:1225] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer d6949d828ece45dcb729633be6d49c97 log is divergent from this leader: its last log entry 4.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20240501 08:14:43.678251 31872 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.678418 31872 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.678548 31872 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.679177 31871 pending_rounds.cc:76] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97: Aborting all ops after (but not including) 4
I20240501 08:14:43.679247 31871 pending_rounds.cc:97] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97: Aborting uncommitted NO_OP operation due to leader change: 4.5
I20240501 08:14:43.679325 31871 raft_consensus.cc:2866] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.681818 31872 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.682139 32000 raft_consensus.cc:1261] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20240501 08:14:43.682264 32000 pending_rounds.cc:76] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf: Aborting all ops after (but not including) 6
I20240501 08:14:43.682348 32000 pending_rounds.cc:97] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20240501 08:14:43.682435 32000 raft_consensus.cc:2866] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240501 08:14:43.683465 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.683676 32260 raft_consensus.cc:3032] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.683573 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.683779 32260 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.683961 32000 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.684056 32260 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.684484 32260 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.685413 31872 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.686452 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.686622 31868 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.686726 31868 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.687230 32260 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.687744 1978 consensus_queue.cc:1035] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 08:14:43.689435 31868 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.690043 31868 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.691100 31807 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.691478 1377 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.691637 1377 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.691569 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.691880 32130 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.691987 32130 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.691896 1377 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.692514 32130 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.693001 32130 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.694023 32002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.694234 32002 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.697057 32130 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.698859 31807 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.699322 2048 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.699470 2048 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.699759 2048 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.702203 32002 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.704667 31868 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.704921 31868 raft_consensus.cc:482] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.705047 31868 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.710188 32000 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.710875 31868 raft_consensus.cc:504] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.711948 31868 leader_election.cc:288] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.712085 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.712321 32260 raft_consensus.cc:3032] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.712227 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90faa3f785f24c53a103ea93a3f405e0" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.712430 32260 raft_consensus.cc:729] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.712479 32130 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.712724 32260 consensus_queue.cc:260] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.713171 32260 raft_consensus.cc:3037] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.713224 31872 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.713505 31872 raft_consensus.cc:482] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.713680 31872 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.715338 32130 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.715700 31807 leader_election.cc:302] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.716064 2148 raft_consensus.cc:2781] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.716147 32260 raft_consensus.cc:2454] T 90faa3f785f24c53a103ea93a3f405e0 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.716367 2148 raft_consensus.cc:686] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.718173 2148 consensus_queue.cc:237] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.717533 31872 raft_consensus.cc:504] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.720786 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.720994 32130 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.721030 31872 leader_election.cc:288] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.721061 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f075dd8fe7f74d108cd84f578643e94d" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.721236 32260 raft_consensus.cc:3032] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.721338 32260 raft_consensus.cc:729] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.721611 32260 consensus_queue.cc:260] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.721917 32260 raft_consensus.cc:3037] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.722380 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.722674 31869 raft_consensus.cc:482] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.722926 31869 raft_consensus.cc:3037] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.723871 32130 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:43.724757 32260 raft_consensus.cc:2396] T f075dd8fe7f74d108cd84f578643e94d P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:43.725198 31807 leader_election.cc:302] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:43.726315 31869 raft_consensus.cc:504] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.726810 2148 raft_consensus.cc:2726] T f075dd8fe7f74d108cd84f578643e94d P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240501 08:14:43.730032 31727 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 9 to 10, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.734254 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.734539 32260 raft_consensus.cc:2379] T 40d7e0033de04a42809bac000fda2a81 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 in current term 9: Already voted for candidate 64f7786244bd4f90af1ad476966482c4 in this term.
I20240501 08:14:43.735595 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d7e0033de04a42809bac000fda2a81" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.736600 31727 catalog_manager.cc:5484] T 90faa3f785f24c53a103ea93a3f405e0 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 9 to 10, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.737144 31727 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 9 to 10, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:43.738586 32130 raft_consensus.cc:2454] T 40d7e0033de04a42809bac000fda2a81 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 9.
I20240501 08:14:43.739709 31807 leader_election.cc:302] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:43.739989 2148 raft_consensus.cc:2781] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.740159 2148 raft_consensus.cc:686] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [term 9 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.740465 2148 consensus_queue.cc:237] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.742249 31869 leader_election.cc:288] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.747063 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.747391 31869 raft_consensus.cc:482] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.747552 31869 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.750386 31869 raft_consensus.cc:504] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.751674 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.751905 32130 raft_consensus.cc:3032] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.751997 32130 raft_consensus.cc:729] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.751847 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae6229e15d2b4adf8f25df4078b2f533" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.752210 32000 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.752274 32130 consensus_queue.cc:260] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.752859 32130 raft_consensus.cc:3037] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.754467 31727 catalog_manager.cc:5484] T 40d7e0033de04a42809bac000fda2a81 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 8 to 9, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 9 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.755024 32000 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.755496 31807 leader_election.cc:302] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.755668 32130 raft_consensus.cc:2454] T ae6229e15d2b4adf8f25df4078b2f533 P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.755821 2148 raft_consensus.cc:2781] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.755960 2148 raft_consensus.cc:686] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.756222 2148 consensus_queue.cc:237] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.757730 31869 leader_election.cc:288] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.760805 31723 catalog_manager.cc:5484] T ae6229e15d2b4adf8f25df4078b2f533 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 9 to 10, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.765309 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.768440 31869 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.768610 31869 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.771539 31869 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.772858 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.773133 32130 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.775914 32130 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.775971 31869 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.776073 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.776329 32000 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.776419 31807 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.776723 1377 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.776891 1377 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.777168 1377 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.779026 32000 raft_consensus.cc:2396] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240501 08:14:43.782074 31727 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 8 to 10, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.783236 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.783495 31869 raft_consensus.cc:482] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.783638 31869 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.786731 31869 raft_consensus.cc:504] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.787550 31869 leader_election.cc:288] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.788112 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.788118 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617ca585bf1845f78b450d28ac3f36fd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.788411 32130 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.788560 32260 raft_consensus.cc:3032] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.788697 32260 raft_consensus.cc:729] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.789005 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.789001 32260 consensus_queue.cc:260] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.789237 31869 raft_consensus.cc:482] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.789307 32260 raft_consensus.cc:3037] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.789371 31869 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.791822 32130 raft_consensus.cc:2454] T 617ca585bf1845f78b450d28ac3f36fd P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.792024 32260 raft_consensus.cc:2396] T 617ca585bf1845f78b450d28ac3f36fd P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20240501 08:14:43.793296 31807 leader_election.cc:302] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:43.793934 1377 raft_consensus.cc:2781] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.794157 1377 raft_consensus.cc:686] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.794987 1377 consensus_queue.cc:237] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.798434 31727 catalog_manager.cc:5484] T 617ca585bf1845f78b450d28ac3f36fd P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 9 to 10, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.801826 31869 raft_consensus.cc:504] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.802615 31869 leader_election.cc:288] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.803666 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.803727 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f80695b5a9f4832b1eac49f4eaeef5e" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.803905 32260 raft_consensus.cc:3032] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.803925 32000 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.804013 32260 raft_consensus.cc:729] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.804280 32260 consensus_queue.cc:260] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.804637 32260 raft_consensus.cc:3037] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.806727 32000 raft_consensus.cc:2454] T 4f80695b5a9f4832b1eac49f4eaeef5e P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 9.
I20240501 08:14:43.807412 32260 raft_consensus.cc:2396] T 4f80695b5a9f4832b1eac49f4eaeef5e P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 08:14:43.808043 31807 leader_election.cc:302] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.808400 1377 raft_consensus.cc:2781] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.808563 1377 raft_consensus.cc:686] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [term 9 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.808848 1377 consensus_queue.cc:237] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.808825 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.809684 31869 raft_consensus.cc:482] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.809885 31869 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.812355 31723 catalog_manager.cc:5484] T 4f80695b5a9f4832b1eac49f4eaeef5e P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 8 to 9, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 9 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.812927 31869 raft_consensus.cc:504] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.814000 31869 leader_election.cc:288] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.813995 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.814218 32130 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.815253 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b96418c77b6483dba686be41d8f5934" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.815449 32260 raft_consensus.cc:3032] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.815550 32260 raft_consensus.cc:729] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.815804 32260 consensus_queue.cc:260] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.816124 32260 raft_consensus.cc:3037] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.816908 32130 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:43.817191 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.817396 31869 raft_consensus.cc:482] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.817507 31869 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.818921 32260 raft_consensus.cc:2396] T 2b96418c77b6483dba686be41d8f5934 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240501 08:14:43.819778 31807 leader_election.cc:302] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 4827555155084237b0199283e5896782, 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:43.820253 1377 raft_consensus.cc:2726] T 2b96418c77b6483dba686be41d8f5934 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20240501 08:14:43.820185 31869 raft_consensus.cc:504] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.820948 31869 leader_election.cc:288] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.821379 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.821576 32260 raft_consensus.cc:3032] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.821565 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.821684 32260 raft_consensus.cc:729] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.821826 32000 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.822005 32260 consensus_queue.cc:260] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.822468 32260 raft_consensus.cc:3037] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.824744 32000 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 9.
I20240501 08:14:43.825773 31807 leader_election.cc:302] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.826134 1377 raft_consensus.cc:2781] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.826349 1377 raft_consensus.cc:686] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [term 9 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.826385 32260 raft_consensus.cc:2454] T d751eea1a7344c2f80ca2fa033d0ee64 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 9.
I20240501 08:14:43.826627 1377 consensus_queue.cc:237] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.826709 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.826958 31869 raft_consensus.cc:482] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.827072 31869 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.829526 31869 raft_consensus.cc:504] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.829901 31723 catalog_manager.cc:5484] T d751eea1a7344c2f80ca2fa033d0ee64 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 8 to 9, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 9 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.830459 31869 leader_election.cc:288] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 8 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.830538 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.830756 32000 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.833124 32000 raft_consensus.cc:2396] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 08:14:43.834164 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.834355 31869 raft_consensus.cc:482] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.834491 31869 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.834901 32000 raft_consensus.cc:1226] T d5eb3eb4e16a4caeb60c74af37968e9c P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Rejecting Update request from peer 64f7786244bd4f90af1ad476966482c4 for earlier term 7. Current term is 8. Ops: []
I20240501 08:14:43.835413 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5eb3eb4e16a4caeb60c74af37968e9c" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.835649 2132 consensus_queue.cc:1046] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.835875 2132 consensus_queue.cc:1225] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Peer 6b7faa87de354980a76c80fb864f9edf log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20240501 08:14:43.836102 32260 raft_consensus.cc:3032] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:43.836201 32260 raft_consensus.cc:729] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.836474 32260 consensus_queue.cc:260] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.836753 32260 raft_consensus.cc:3037] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.837433 31869 raft_consensus.cc:504] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.838433 31869 leader_election.cc:288] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 8 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.840257 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.840409 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aac3040ec2f4c2185a48967e3c245ee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.840493 32130 raft_consensus.cc:3032] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 7 LEADER]: Stepping down as leader of term 7
I20240501 08:14:43.840605 32130 raft_consensus.cc:729] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4827555155084237b0199283e5896782, State: Running, Role: LEADER
I20240501 08:14:43.840626 32000 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.840977 32260 raft_consensus.cc:2396] T d5eb3eb4e16a4caeb60c74af37968e9c P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20240501 08:14:43.840869 32130 consensus_queue.cc:260] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
W20240501 08:14:43.841127 1978 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20240501 08:14:43.841262 32130 raft_consensus.cc:3037] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 7 FOLLOWER]: Advancing to term 8
I20240501 08:14:43.841552 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.841851 31869 raft_consensus.cc:482] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.841881 31807 leader_election.cc:302] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.842069 31869 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.842348 2148 raft_consensus.cc:2726] T d5eb3eb4e16a4caeb60c74af37968e9c P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20240501 08:14:43.843133 32000 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 8.
I20240501 08:14:43.843501 31807 leader_election.cc:302] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.843786 2148 raft_consensus.cc:2781] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Leader election won for term 8
I20240501 08:14:43.843951 2148 raft_consensus.cc:686] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [term 8 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.844158 32130 raft_consensus.cc:2454] T 5aac3040ec2f4c2185a48967e3c245ee P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 8.
I20240501 08:14:43.844234 2148 consensus_queue.cc:237] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.845845 31869 raft_consensus.cc:504] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.846442 31869 leader_election.cc:288] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.847189 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.847252 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d92472b3840640e99f9af90c05ae7cee" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.847460 32000 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.847479 32260 raft_consensus.cc:3032] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.847604 32260 raft_consensus.cc:729] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.847942 32260 consensus_queue.cc:260] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.848362 32260 raft_consensus.cc:3037] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.848897 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.848443 31723 catalog_manager.cc:5484] T 5aac3040ec2f4c2185a48967e3c245ee P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 7 to 8, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 8 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.849071 31869 raft_consensus.cc:482] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.849174 31869 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.850642 32000 raft_consensus.cc:2454] T d92472b3840640e99f9af90c05ae7cee P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.851274 31807 leader_election.cc:302] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.851899 2148 raft_consensus.cc:2781] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.852077 2148 raft_consensus.cc:686] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.852130 32260 raft_consensus.cc:2396] T d92472b3840640e99f9af90c05ae7cee P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20240501 08:14:43.852356 2148 consensus_queue.cc:237] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.853237 31869 raft_consensus.cc:504] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.853946 31869 leader_election.cc:288] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.854372 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.854570 32130 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.855245 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "93a196489f264058a064dc84690ba947"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.855433 31869 raft_consensus.cc:482] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.855532 31869 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.856514 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e8e84d1538e460ebb9785fca94288bd" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.856765 32000 raft_consensus.cc:3037] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.856961 31723 catalog_manager.cc:5484] T d92472b3840640e99f9af90c05ae7cee P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 9 to 10, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.859256 32000 raft_consensus.cc:2396] T 1e8e84d1538e460ebb9785fca94288bd P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 08:14:43.860993 32130 raft_consensus.cc:2454] T 1e8e84d1538e460ebb9785fca94288bd P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.861471 31807 leader_election.cc:302] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.860985 31869 raft_consensus.cc:504] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.862043 2148 raft_consensus.cc:2781] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.862205 2148 raft_consensus.cc:686] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.862215 31869 leader_election.cc:288] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.862540 2148 consensus_queue.cc:237] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.862758 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.862983 32130 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.864526 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.864768 31869 raft_consensus.cc:482] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.864961 31869 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.865589 32130 raft_consensus.cc:2454] T 93a196489f264058a064dc84690ba947 P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.865949 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93a196489f264058a064dc84690ba947" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.866149 32000 raft_consensus.cc:3037] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.867312 31869 raft_consensus.cc:504] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.867954 31869 leader_election.cc:288] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.868271 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.868502 32260 raft_consensus.cc:3032] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.868577 32260 raft_consensus.cc:729] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.868798 32260 consensus_queue.cc:260] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.869004 32000 raft_consensus.cc:2396] T 93a196489f264058a064dc84690ba947 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20240501 08:14:43.869061 32260 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.870977 32260 raft_consensus.cc:2396] T cef61d27228f43cfbae61993aee7f432 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20240501 08:14:43.871747 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef61d27228f43cfbae61993aee7f432" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.871969 32000 raft_consensus.cc:3037] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.872017 31807 leader_election.cc:302] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.872457 2148 raft_consensus.cc:2781] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.872612 2148 raft_consensus.cc:686] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.872916 2148 consensus_queue.cc:237] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.873412 31723 catalog_manager.cc:5484] T 1e8e84d1538e460ebb9785fca94288bd P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 8 to 10, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.874702 32000 raft_consensus.cc:2454] T cef61d27228f43cfbae61993aee7f432 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.875176 31807 leader_election.cc:302] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters: 64f7786244bd4f90af1ad476966482c4
I20240501 08:14:43.875427 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.875525 1377 raft_consensus.cc:2781] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.875633 31869 raft_consensus.cc:482] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.875671 1377 raft_consensus.cc:686] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.875751 31869 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.875914 1377 consensus_queue.cc:237] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.878713 31869 raft_consensus.cc:504] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.879350 31869 leader_election.cc:288] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.880085 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.880331 32260 raft_consensus.cc:3032] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.880437 32260 raft_consensus.cc:729] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.880347 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61369fbeb41a49588907104bcfd6fa0f" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.880594 32130 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.880746 32260 consensus_queue.cc:260] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.881065 32260 raft_consensus.cc:3037] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.880940 31723 catalog_manager.cc:5484] T 93a196489f264058a064dc84690ba947 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 8 to 10, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:43.881589 31723 catalog_manager.cc:5484] T cef61d27228f43cfbae61993aee7f432 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 9 to 10, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.883683 32130 raft_consensus.cc:2454] T 61369fbeb41a49588907104bcfd6fa0f P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 9.
I20240501 08:14:43.883775 32260 raft_consensus.cc:2396] T 61369fbeb41a49588907104bcfd6fa0f P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240501 08:14:43.884147 31807 leader_election.cc:302] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.884220 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.884466 31869 raft_consensus.cc:482] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.884472 2148 raft_consensus.cc:2781] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.884617 31869 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.884644 2148 raft_consensus.cc:686] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [term 9 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.884887 2148 consensus_queue.cc:237] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.887569 31869 raft_consensus.cc:504] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.888267 31869 leader_election.cc:288] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Requested vote from peers 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.888435 31723 catalog_manager.cc:5484] T 61369fbeb41a49588907104bcfd6fa0f P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 8 to 9, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 9 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.888798 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.889016 32000 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.889046 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b50688fadbf64a588bb2a9cba14024f5" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.889263 32130 raft_consensus.cc:3037] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.890187 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.890491 31869 raft_consensus.cc:482] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.890664 31869 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.892992 32000 raft_consensus.cc:2396] T b50688fadbf64a588bb2a9cba14024f5 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20240501 08:14:43.893023 31869 raft_consensus.cc:504] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.893154 32130 raft_consensus.cc:2454] T b50688fadbf64a588bb2a9cba14024f5 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 9.
I20240501 08:14:43.893496 31807 leader_election.cc:302] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters: 6b7faa87de354980a76c80fb864f9edf
I20240501 08:14:43.893745 31869 leader_election.cc:288] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.893816 2048 raft_consensus.cc:2781] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.893988 2048 raft_consensus.cc:686] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [term 9 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.894229 2048 consensus_queue.cc:237] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.894415 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.894614 32132 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.894902 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8011eb1bd4c42ce89517736009576e3" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.895125 32260 raft_consensus.cc:3032] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Stepping down as leader of term 8
I20240501 08:14:43.895220 32260 raft_consensus.cc:729] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.895540 32260 consensus_queue.cc:260] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.895866 32260 raft_consensus.cc:3037] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 8 FOLLOWER]: Advancing to term 9
I20240501 08:14:43.897500 32132 raft_consensus.cc:2454] T b8011eb1bd4c42ce89517736009576e3 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 9.
I20240501 08:14:43.897939 31807 leader_election.cc:302] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.898237 2148 raft_consensus.cc:2781] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Leader election won for term 9
I20240501 08:14:43.898412 2148 raft_consensus.cc:686] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [term 9 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.898627 32260 raft_consensus.cc:2396] T b8011eb1bd4c42ce89517736009576e3 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20240501 08:14:43.898689 2148 consensus_queue.cc:237] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.900182 31869 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79"
dest_uuid: "d6949d828ece45dcb729633be6d49c97"
from {username='slave'} at 127.0.0.1:33368
I20240501 08:14:43.900372 31869 raft_consensus.cc:482] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.900502 31869 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.902128 31723 catalog_manager.cc:5484] T b50688fadbf64a588bb2a9cba14024f5 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 7 to 9, leader changed from 6b7faa87de354980a76c80fb864f9edf (127.6.56.66) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 9 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:43.902902 31869 raft_consensus.cc:504] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.903499 31869 leader_election.cc:288] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), 6b7faa87de354980a76c80fb864f9edf (127.6.56.66:34121)
I20240501 08:14:43.904320 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.904402 32000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0448f8ee0f7a4932932247445b361d79" candidate_uuid: "d6949d828ece45dcb729633be6d49c97" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
I20240501 08:14:43.904528 32260 raft_consensus.cc:3032] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.904671 32000 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.904628 32260 raft_consensus.cc:729] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 64f7786244bd4f90af1ad476966482c4, State: Running, Role: LEADER
I20240501 08:14:43.905689 32260 consensus_queue.cc:260] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.906018 32260 raft_consensus.cc:3037] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.906940 32000 raft_consensus.cc:2454] T 0448f8ee0f7a4932932247445b361d79 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6949d828ece45dcb729633be6d49c97 in term 10.
I20240501 08:14:43.907402 31807 leader_election.cc:302] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b7faa87de354980a76c80fb864f9edf, d6949d828ece45dcb729633be6d49c97; no voters:
I20240501 08:14:43.907732 2148 raft_consensus.cc:2781] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.907864 2148 raft_consensus.cc:686] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Leader. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.908145 2148 consensus_queue.cc:237] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.910071 32260 raft_consensus.cc:2396] T 0448f8ee0f7a4932932247445b361d79 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d6949d828ece45dcb729633be6d49c97 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20240501 08:14:43.912858 31723 catalog_manager.cc:5484] T b8011eb1bd4c42ce89517736009576e3 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 8 to 9, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 9 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.915043 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:43.915241 32000 raft_consensus.cc:482] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.915485 32000 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.918377 32000 raft_consensus.cc:504] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.919324 31723 catalog_manager.cc:5484] T 0448f8ee0f7a4932932247445b361d79 P d6949d828ece45dcb729633be6d49c97 reported cstate change: term changed from 9 to 10, leader changed from 64f7786244bd4f90af1ad476966482c4 (127.6.56.68) to d6949d828ece45dcb729633be6d49c97 (127.6.56.65). New cstate: current_term: 10 leader_uuid: "d6949d828ece45dcb729633be6d49c97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.919639 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.919860 32130 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.922435 32130 raft_consensus.cc:2396] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 4827555155084237b0199283e5896782 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20240501 08:14:43.923476 32000 leader_election.cc:288] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 11 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.924010 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccfa1910629f47c6b3b6caf6c1a8c3f1" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.924229 32260 raft_consensus.cc:3037] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.924774 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:43.925038 32000 raft_consensus.cc:482] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.925244 32000 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.926955 32260 raft_consensus.cc:2454] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 64f7786244bd4f90af1ad476966482c4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 11.
I20240501 08:14:43.927448 31938 leader_election.cc:302] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters: 4827555155084237b0199283e5896782
I20240501 08:14:43.927821 2074 raft_consensus.cc:2781] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 11 FOLLOWER]: Leader election won for term 11
I20240501 08:14:43.928025 2074 raft_consensus.cc:686] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [term 11 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.928113 32000 raft_consensus.cc:504] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.928426 2074 consensus_queue.cc:237] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.928743 32000 leader_election.cc:288] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 11 election: Requested vote from peers d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515), 4827555155084237b0199283e5896782 (127.6.56.67:41473)
I20240501 08:14:43.929183 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.929427 31869 raft_consensus.cc:3032] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Stepping down as leader of term 10
I20240501 08:14:43.929515 31869 raft_consensus.cc:729] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.929520 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7795f40f344b2e92dfc60a5ff1173a" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.929733 32130 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.929812 31869 consensus_queue.cc:260] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.930112 31869 raft_consensus.cc:3037] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.932562 32130 raft_consensus.cc:2454] T 4e7795f40f344b2e92dfc60a5ff1173a P 4827555155084237b0199283e5896782 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 11.
I20240501 08:14:43.933009 31869 raft_consensus.cc:2396] T 4e7795f40f344b2e92dfc60a5ff1173a P d6949d828ece45dcb729633be6d49c97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20240501 08:14:43.933032 31938 leader_election.cc:302] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:43.933344 2074 raft_consensus.cc:2781] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 11 FOLLOWER]: Leader election won for term 11
I20240501 08:14:43.933493 2074 raft_consensus.cc:686] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [term 11 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.933709 2074 consensus_queue.cc:237] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } }
I20240501 08:14:43.935132 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:43.935317 32000 raft_consensus.cc:482] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.935437 32000 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.937027 31723 catalog_manager.cc:5484] T ccfa1910629f47c6b3b6caf6c1a8c3f1 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 9 to 11, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 11 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.937944 32000 raft_consensus.cc:504] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.939424 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.939663 32260 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.942281 32260 raft_consensus.cc:2454] T 73d13200a9464963bf7501faad8cd6f0 P 64f7786244bd4f90af1ad476966482c4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 11.
I20240501 08:14:43.942749 32000 leader_election.cc:288] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 11 election: Requested vote from peers 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.942495 31727 catalog_manager.cc:5484] T 4e7795f40f344b2e92dfc60a5ff1173a P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 10 to 11, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 11 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } }
I20240501 08:14:43.942979 31938 leader_election.cc:302] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:43.943262 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d13200a9464963bf7501faad8cd6f0" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.943395 2074 raft_consensus.cc:2781] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 11 FOLLOWER]: Leader election won for term 11
I20240501 08:14:43.943504 31869 raft_consensus.cc:3032] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Stepping down as leader of term 10
I20240501 08:14:43.943595 31869 raft_consensus.cc:729] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.943794 2074 raft_consensus.cc:686] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [term 11 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.943882 31869 consensus_queue.cc:260] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.944195 31869 raft_consensus.cc:3037] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.944239 2074 consensus_queue.cc:237] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.948323 31727 catalog_manager.cc:5484] T 73d13200a9464963bf7501faad8cd6f0 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 10 to 11, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 11 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } }
I20240501 08:14:43.949770 32130 raft_consensus.cc:1261] T 1e872bea48fc479686691121187aa0db P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240501 08:14:43.949903 31869 raft_consensus.cc:2396] T 73d13200a9464963bf7501faad8cd6f0 P d6949d828ece45dcb729633be6d49c97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20240501 08:14:43.950250 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:43.950510 32000 raft_consensus.cc:482] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.950636 32000 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.950680 1978 consensus_queue.cc:1035] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.954177 32002 raft_consensus.cc:1261] T 1e872bea48fc479686691121187aa0db P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Refusing update from remote peer 64f7786244bd4f90af1ad476966482c4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20240501 08:14:43.954690 2132 consensus_queue.cc:1035] T 1e872bea48fc479686691121187aa0db P 64f7786244bd4f90af1ad476966482c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 08:14:43.955024 32000 raft_consensus.cc:504] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.957374 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.957556 32260 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.956346 32000 leader_election.cc:288] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.959853 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:43.960062 32260 raft_consensus.cc:2454] T 5634d60715fc486b81d38f1dbad590e4 P 64f7786244bd4f90af1ad476966482c4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 10.
I20240501 08:14:43.960120 32000 raft_consensus.cc:482] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.960269 32000 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.960453 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5634d60715fc486b81d38f1dbad590e4" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.960549 31938 leader_election.cc:302] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64f7786244bd4f90af1ad476966482c4, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:43.960676 32130 raft_consensus.cc:3037] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.960845 2067 raft_consensus.cc:2781] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.961059 2067 raft_consensus.cc:686] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [term 10 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.961365 2067 consensus_queue.cc:237] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } }
I20240501 08:14:43.964972 32130 raft_consensus.cc:2396] T 5634d60715fc486b81d38f1dbad590e4 P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20240501 08:14:43.964995 32000 raft_consensus.cc:504] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.965415 31723 catalog_manager.cc:5484] T 5634d60715fc486b81d38f1dbad590e4 P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 8 to 10, leader changed from 4827555155084237b0199283e5896782 (127.6.56.67) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 10 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } }
I20240501 08:14:43.965974 32000 leader_election.cc:288] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 11 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.966194 32130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.966382 32130 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.966313 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c1f781d8e3645e0b35d0be82f6880cf" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.966593 31869 raft_consensus.cc:3032] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Stepping down as leader of term 10
I20240501 08:14:43.966686 31869 raft_consensus.cc:729] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.966961 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:43.967111 32002 raft_consensus.cc:482] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.967183 32002 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.967052 31869 consensus_queue.cc:260] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.967428 31869 raft_consensus.cc:3037] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Advancing to term 11
I20240501 08:14:43.969509 32002 raft_consensus.cc:504] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.970574 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.970813 32132 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.970888 32130 raft_consensus.cc:2454] T 9c1f781d8e3645e0b35d0be82f6880cf P 4827555155084237b0199283e5896782 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 11.
I20240501 08:14:43.971818 31869 raft_consensus.cc:2396] T 9c1f781d8e3645e0b35d0be82f6880cf P d6949d828ece45dcb729633be6d49c97 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20240501 08:14:43.973212 31938 leader_election.cc:302] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters: d6949d828ece45dcb729633be6d49c97
I20240501 08:14:43.973649 2074 raft_consensus.cc:2781] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 11 FOLLOWER]: Leader election won for term 11
I20240501 08:14:43.973814 2074 raft_consensus.cc:686] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [term 11 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.974092 2074 consensus_queue.cc:237] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } }
I20240501 08:14:43.976891 32132 raft_consensus.cc:2454] T 9fe2118801714da99fe7f3c0585e57fc P 4827555155084237b0199283e5896782 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b7faa87de354980a76c80fb864f9edf in term 10.
I20240501 08:14:43.977329 31938 leader_election.cc:302] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4827555155084237b0199283e5896782, 6b7faa87de354980a76c80fb864f9edf; no voters:
I20240501 08:14:43.978225 2074 raft_consensus.cc:2781] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Leader election won for term 10
I20240501 08:14:43.978374 2074 raft_consensus.cc:686] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [term 10 LEADER]: Becoming Leader. State: Replica: 6b7faa87de354980a76c80fb864f9edf, State: Running, Role: LEADER
I20240501 08:14:43.978490 31869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fe2118801714da99fe7f3c0585e57fc" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "d6949d828ece45dcb729633be6d49c97"
I20240501 08:14:43.978731 31869 raft_consensus.cc:3032] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 9 LEADER]: Stepping down as leader of term 9
I20240501 08:14:43.978631 2074 consensus_queue.cc:237] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.978821 31869 raft_consensus.cc:729] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d6949d828ece45dcb729633be6d49c97, State: Running, Role: LEADER
I20240501 08:14:43.979120 31869 consensus_queue.cc:260] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.979439 31869 raft_consensus.cc:3037] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.977355 32002 leader_election.cc:288] T 9fe2118801714da99fe7f3c0585e57fc P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), d6949d828ece45dcb729633be6d49c97 (127.6.56.65:38515)
I20240501 08:14:43.980654 32002 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'} at 127.0.0.1:52718
I20240501 08:14:43.980862 32002 raft_consensus.cc:482] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 08:14:43.980983 32002 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.981760 31869 raft_consensus.cc:2396] T 9fe2118801714da99fe7f3c0585e57fc P d6949d828ece45dcb729633be6d49c97 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7faa87de354980a76c80fb864f9edf for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20240501 08:14:43.983098 32002 raft_consensus.cc:504] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "64f7786244bd4f90af1ad476966482c4" member_type: VOTER last_known_addr { host: "127.6.56.68" port: 36729 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } }
I20240501 08:14:43.983383 31723 catalog_manager.cc:5484] T 9c1f781d8e3645e0b35d0be82f6880cf P 6b7faa87de354980a76c80fb864f9edf reported cstate change: term changed from 10 to 11, leader changed from d6949d828ece45dcb729633be6d49c97 (127.6.56.65) to 6b7faa87de354980a76c80fb864f9edf (127.6.56.66). New cstate: current_term: 11 leader_uuid: "6b7faa87de354980a76c80fb864f9edf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4827555155084237b0199283e5896782" member_type: VOTER last_known_addr { host: "127.6.56.67" port: 41473 } } peers { permanent_uuid: "6b7faa87de354980a76c80fb864f9edf" member_type: VOTER last_known_addr { host: "127.6.56.66" port: 34121 } } peers { permanent_uuid: "d6949d828ece45dcb729633be6d49c97" member_type: VOTER last_known_addr { host: "127.6.56.65" port: 38515 } } }
I20240501 08:14:43.984117 32002 leader_election.cc:288] T a9de4d2812b344acb197768825bd4f04 P 6b7faa87de354980a76c80fb864f9edf [CANDIDATE]: Term 10 election: Requested vote from peers 4827555155084237b0199283e5896782 (127.6.56.67:41473), 64f7786244bd4f90af1ad476966482c4 (127.6.56.68:36729)
I20240501 08:14:43.984952 32132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "4827555155084237b0199283e5896782"
I20240501 08:14:43.985165 32132 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 4827555155084237b0199283e5896782 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.984712 32260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9de4d2812b344acb197768825bd4f04" candidate_uuid: "6b7faa87de354980a76c80fb864f9edf" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: true dest_uuid: "64f7786244bd4f90af1ad476966482c4"
I20240501 08:14:43.985360 32260 raft_consensus.cc:3037] T a9de4d2812b344acb197768825bd4f04 P 64f7786244bd4f90af1ad476966482c4 [term 9 FOLLOWER]: Advancing to term 10
I20240501 08:14:43.986613 32000 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d751eea1a7344c2f80ca2fa033d0ee64"
dest_uuid: "6b7faa87de354980a76c80fb864f9edf"
from {username='slave'